Lisa Wade recently made me aware that a group of psychologists have decided to try and verify the results of every article written in three journals in 2008. I think this is a great idea. And the likelihood that the results can be replicated look pretty unlikely. As the Chronicle piece I’ve linked to points out,
Recently, a scientist named C. Glenn Begley attempted to replicate 53 cancer studies he deemed landmark publications. He could only replicate six. Six! Last December I interviewed Christopher Chabris about his paper titled “Most Reported Genetic Associations with General Intelligence Are Probably False Positives.” Most!”
If only 6 out of 53 cancer studies can be replicated, I shudder what to think what the social science data will say (in part because there is so much more noise in a lot of our research design). It’s more of an open question to me what this means. It could mean that researchers are generally frauds. This could take different forms. On one extreme they could simply fabricate results. This happens, perhaps not with amazing regularity. But it’s hard to say because we rarely do replication studies. On the other hand, it could be something less nefarious. Scholars hunt and hunt for a significant result and once they find one that is interesting enough they publish. But such results might be produced more than actually observed. Or for experimental work it could simply mean that the published result (or the replication) are outliers, or that social conditions have changed influencing the likelihood of observing a result (I think of performativity here, particularly for famous results).
Now it may seem fresh to hear an ethnographer bring up the problems of replicability in social science work (although as an aside I’ll point out that I do work no one knows about — experimental work! And I have new such work under review). For an ethnographer dealing with this kind of stuff is tricky. At first I thought I would put my fieldnotes online. But that provided impossible, because there were some terribly personal things in there that could hurt teenagers; further, my notes weren’t anonymized. And so I decided not to make the place of my study anonymous. Here I fall a bit into the Mitch Duneier school, which suggests that anonymity is more a guise to protect the researcher than the subjects. And so you can dig up my subjects. You can go to the school. You can find the newspaper reports of incidents I talk about (the hazing scandal, for one).
In fact, ethnographers have a bit of a history of verifying results. The Duneier/Klinenberg debate would be one example. And in many ways the Chicago School is very much about revisiting and re-examining results, often in the same neighborhoods. I am not arguing here that ethnography is better than other methods at replicating results. I’m simply pointing out that the not uncommon refrain about the lack of replicability of my main kind of work is hardly accurate.
As for other work, I wonder. What if a group of us decided to try and replicate every finding in two major journals published in 2008? We used available data to see if we could produce the same statistical results? We went back to communities to see if what was reported actually happened (a la Duneier). We revisited interviews or historical records. Would we, like the cancer researcher only be able to replicate 6 out of 53 studies. And if that were the case, what would it mean? I would opine that this would be worse for folks using available data, because they would not have the some protections as we experimentalists or ethnographers (the world has changed, one of these results is an outlier, and it could be your replication). But still, it would be an interesting task.