I was fortunate to attend a talk by an economist visiting our campus last week and, during lunch, she mentioned the embarrassment that the Reinhart and Rogoff scandal caused the economics profession, including being flogged by Stephen Colbert. I then explained the embarrassment in our fair discipline, the Regnerus affair, of which she had not heard (which, itself, made me very happy). I realize that many might be losing an appetite for this topic, but I think that juxtaposing these two episodes shows some fairly sharp contrasts and lessons for academic work more generally.
Both, I believe, point to fundamental problems in our publication systems. Equally important, however, I submit that sociology’s handling of the Regnerus affair actually conveys a relatively healthy response that, through the subsequent devastating critiques, produced important knowledge. I also submit that the publication of Regnerus’s paper led to this outcome far quickly than what happened in response to the Reinhart and Rogoff scandal.
Social Science Research published Regnerus paper, with all of its substantial flaws, through the normal peer-review process, with substantial flaws of its own. Then what happened? A flurry of intellectual, political, and legal writing on the subject flourished — almost instantly. Regnerus made his data available. Neal took this data and made a publicly accessible and open source reanalysis repository based on that data, which then became a published critique of Regnerus’ original study. Science moved very fast and showed the flaws of the initial results.1 Any author hoping to cite Regnerus’ findings in the future should now also cite the scientific response.
The process also led to substantial introspection about our discipline’s norms. The journal itself devoted considerable space to its own policies and shortcomings (and the description of the publication process provides an invaluable resource for graduate students who want to learn about reviews, writing, and editing). We had more discussions that I can enumerate that ranged, even this week, from retraction to conflicts of interest to the role of outside groups in scientific research.
The Reinhart & Rogoff paper, by comparison, was not subjected to peer review. Instead, they posted the paper as a working paper at NBER, an insider’s club of economists. It appears that the data were not made public. Upon release, a new paper (itself released as a working paper without peer review), found the original included an Excel (!) formula error and questionable analytical decisions (all with precedent in the literature, however). An additional analysis showed that Reinhart & Rogoff probably got the causal order wrong. The time lapse between the publication of the original and the revision in economics, without peer review: almost three years (a year and a half if you count the last revision). And its not as if their paper didn’t matter, world economies were put in precarious situations partially on the basis of their results.
I think that science worked in both cases. I am proud that sociology made quicker work of debunking incorrect findings compared to economics, and think that the institutional structure of a journal helped sociology move more quickly (NB: I am not proud that bad research got published in the first place). In both cases, however, our intellectual curiosity and academic freedom to pursue new research (and shared data) allowed subsequent research to correct the scientific record.
That said, we must be extra careful when our results affect the lives of our fellow humans through Supreme Court cases or central bank policies. We should strive to publish on topics important enough for people to read our work and to do so in a language that they can understand. Arguing no one reads or cares about our work is a poor defense and an indictment on much of our own research. We must always act as if our research matters (because it does) and in both cases I fear that our disciplines became too cavalier publishing results and not double- and triple-checking our work when the stakes for others’ lives were high.
Science always progresses haltingly. Progress comes from false starts, challenges to received wisdom, and questioned assumptions. If we expect peer review to be perfect every time or no bad research to be published we not only set ourselves against an impossible standard, which is a danger, but because we would expect everything to be flawless we would stifle innovative research and risk becoming beholden to received wisdom. That, I believe, is a greater danger.
I note that the Perrin, Cohen, and Caren piece went from submitted to accepted in eight and a half weeks, two of which included Thanksgiving and Christmas/New Year’s. This seems like a comparable timeline to the Regnerus publication, but I do not know the normal publication time at Journal of Gay & Lesbian Mental Health, so it might be that this journal is just extremely prompt. I would value any feedback from the authors, especially since two of whom are contributors here, about their experience publishing their piece. ↩