when to retract?

The Regnerus controversy has us debating, among other things, the criteria for retracting a published sociology paper from a journal. There are clearly some cases in which there is widespread agreement that a retraction is warranted:

  1. fraudulent data
  2. plagiarism
  3. a mistake that invalidates the analysis

The case of Regnerus, however, has us disagreeing. So far, I can see that there is little agreement over whether these criteria require a retraction:

  1. a paper whose review included an undisclosed conflict of interest
  2. a paper with findings that have been demonstrated to be incorrect
  3. a paper of poor quality (such that it should not have been published in the first place)
  4. a paper with errant findings that is interacting in negative ways with the public/policy spheres

So far, Andy, Neal, Phil, Fabio, and Jeremy have made their thoughts known. I wonder if we can have a productive discussion about where the line–or at least the grey area–should be. Leaving it up to the editors without clear disciplinary norms seems less than optimal.

33 thoughts on “when to retract?”

  1. Thanks for starting this off, Tina. I suppose another way of asking it might be: what purpose is served by not retracting it? And since COPE’s standard is “publications that contain such seriously flawed or erroneous data that their findings and conclusions cannot be relied upon,” how is that standard not met?


  2. I could be wrong of course, but I’m not convinced that one can, or should, create anything but very general broad guidelines, such as those already covered by 1-3. A case-by-case approach may be better suited for situations beyond that, allowing current scientific and/or research norms (whatever they may be at the time) to dictate the boundaries/guidelines/yardsticks of the debate. Besides that, perhaps changing #1 to simply “fraud” in any form would be appropriate, to cover more than simply fraudulent data. Not sure what that would be precisely, but I can imagine interesting scenarios, especially in sociology.


  3. What about if the editor or editorial board just decide that want a do-over on the reviews? They chose bad reviewers and a bad piece of research got through, or the editor was irresponsible and the editorial board wants to reconsider. It’s a shame, maybe an archaic courtesy of printed-paper days, that a paper has to remain “published” forever.

    Of course, Pandora’s box, witch-hunts, petitions, etc. – no fun for anyone. But *if* you could do it right, what’s wrong with that?

    If we invented the system today, wouldn’t we have toward a system of rolling, interactive peer-reviews, including a post-“publication” component? (e.g., http://futureofscipub.wordpress.com/open-post-publication-peer-review/)


    1. But we already have a system that allows flaws in published work to be identified and corrected in subsequent work. This system could function better if we (a) required authors to make their data and code publicly available (as is happening in many other fields), and (b) didn’t turn up our collective noses at replication studies. But these are minor tweaks to an existing system, not wholesale shifts in the review process.

      All else being equal, it seems far more productive for us to spend our time doing research that corrects/extends/replicates already-published work than it is writing “do-over” reviews. (Did you read Wright’s comments about how many people he typicaly has to ask to get three reviews? I knew it was bad, but not that bad!)

      Another cost of a redo system is that editors would have to adjudicate between valid requests to redo reviews and “witch-hunts” or other politically (or personally) motivated efforts to discredit an author. A secondary consequence, I’d guess, is a decrease in the number and status of scholars who are willing to edit journals. It’s a thankless task as it is.


      1. Regarding Wright’s comment on how many people he has to ask before getting a good review… If the editorial board insists on having the power to “edit reviews” then they are going to have a hard time getting reviewers. I turned them down because of that very issue. Really out of bounds and borderline unethical on that issue.


  4. I think if a paper has been procedurally properly reviewed, it’s hard for me to see a retraction for something that was “in plain sight” to the reviewers, unless perhaps it was an unambiguous technical error of some sort. (Maybe a guide to whether work meets this standard of error is to consider whether the reviewers themselves support retraction.)

    Specifically, I don’t think retraction applies to work that is simply shown later to be wrong. Lots of scientific articles end up being wrong, even having problems that seem dumb that peer reviewers did not point out. (The Duncan Watts observation that everything-seems-obvious-in-retrospect sort of applies here.) Retraction is the academic version of annulment; I take it to mean the journal recants having ever published the article in the first place.

    In the case of a review process that is subsequently revealed to have been improper, I could see a journal disclaiming that an article should no longer be considered to have been peer-reviewed to the standards of that journal. I’m not sure about the concept of “re-doing” reviews, but I can see where a paper that was improperly reviewed and subsequently revealed to have gross flaws would be an especial candidate for withdrawing its status as having been peer-reviewed. Not sure if that’s the same as retraction.

    (Also, I’m not going to claim to have enough command offhand of the details about peer review in the Regnerus case that have been raised, so I’m not making any statement about whether I’d agree with saying that paper should no longer be considered to have been peer reviewed.)


    1. jeremy, “I could see a journal disclaiming that an article should no longer be considered to have been peer-reviewed to the standards of that journal. I’m not sure about the concept of “re-doing” reviews, but I can see where a paper that was improperly reviewed and subsequently revealed to have gross flaws would be an especial candidate for withdrawing its status as having been peer-reviewed. Not sure if that’s the same as retraction.”

      I would be FINE with this. Perfectly FINE.


  5. I like the way Tina has framed this.

    One complicating factor in this case is that SSR is owned and operated by Elsevier. This is in contrast to almost all the other research journals which are run by associations (e.g. ASR, Demography) or universities (e.g. AJS, Social Forces). So even if a broad group of scholars comes to a consensus that the peer-reviewed status should be withdrawn, that isn’t likely to influence Elsevier. I know nothing about the journal’s structure, but I would guess being on the Board of Advisory Editors just means that you are asked to review a couple of times a year and that you get a thank you letter every now and then. I could be wrong, but I highly doubt they meet and discuss journal policy issues.

    Considering the journal has made only cosmetic changes in the last year (an additional sentence about conflicts in the reviewer letters), my guess is that a public, mass resignation of board members is just about the only thing that is likely to get on their radar. Maybe a COPE letter, but that seems highly unlikely to happen given the jurisdiction issues. Maybe a huge boycott? Perhaps Brayden could chime in with some evidence based tactics that are likely to work against corporate targets. Shareholder resolution anyone?

    From a practical standpoint, I think this means that the bar for getting a retraction is quite high. This is not a situation where both sides could argue in front of some sort of neutral, fact finding arbitration board. I think the default response of many scholars (including myself) is that bad science should normally be refuted with good science, not retraction.

    My opinion is that the an argument based on a fundamentally flawed and biased review process has the best shot of convincing board members to request a retraction or peer-review withdrawal. Say for instance, it turned out Regnerus handpicked his set of reviewers, and a majority of them had pre-existing ties to the project. That would seem to threaten the legitimacy of the entire journal–what other authors were using that trick?

    While we have more public information about this review process than that of maybe any other sociology article, I’m not sure there is enough evidence in the public domain yet to make this review-based argument persuasive enough to actual win. This evidence may never materialize; either because the theory is wrong or the flow of NFSS related documents dries up.


  6. A couple of quick things to say:
    1.) There are two COPEs: one is ASA’s Committee on Professional Ethics, which clearly has jurisdiction issues. The other is the Committee on Publication Ethics, a large organization comprising many journals including those owned by Elsevier (see this page.)

    2.) Elsevier’s process says that a retracted article will remain available but behind an alert page saying it’s been retracted. So any historical value in the fact that bad research was once published is maintained.

    3.) Elsevier’s process also says it’s up to the editor of the journal to decide to retract.

    4.) I still think the situation as it stands meets the stated criteria for retraction.


    1. I agree that further persuasion is unlikely to work in the absence of new information or a movement strategic breakthrough of some kind (Organized boycott, with pledges not to submit or review published prominently?). Or the news could change (French mobs marching in Regnerus t-shirts, Scalia citing it in a gruesome majority opinion.)


      1. I’m very intrigued by the threats/promises of boycotts of journals in general. Obviously, the conflict over this particular paper is only one cause spurring talk of boycotts (egregious subscription fees seems to be more powerful as a motivator.) I’m not aware of any boycotts that have taken off or had any substantial traction.
        It does sound, however, like many academics have undertaken personal boycotts of reviewing–although this doesn’t seem tied to any journal practice (mostly, see sreliason above) aside from asking for reviews.


      2. Regarding boycotting journals, there are a lot of good reasons academics of conscience should be boycotting Elsevier journals across the board, including its practice of buying up good journals founded by others and then turning them into profit cows, its practice of predatory pricing and sucking up money from library budgets, and its practice of creating pseudo-journals full of industry-produced “research” and pretending that it is peer-reviewed. Unlike many other fields, sociology so far has not lost most of its journals to Elsevier and can mostly just refuse to send papers to Elsevier journals or refuse to review for them without much threat of losing publication outlets important to professional advancement.


      3. Re “profit cows”: In 1999, the year before Elsevier purchased Social Science Research, SSR published 427 pages. Last year SSR published 1628 pages. The price that publishers charge libraries depends on a lot of things, but the number of pages is usually in the equation.


      4. The listed electronic/print institutional subscription price of Social Science Research is $863. The listed equivalent institutional subscription price for the first non-ASA SAGE journal that springs to mind, Sociological Methods and Research, is $964. SSR publishes a lot more pages.

        It’s an interesting counterfactual to imagine how all this might have played out if SSR had been a SAGE journal rather than an Elsevier journal. (For any spectators, sociologists are in bed with SAGE as a result of ASA journals being published through a partnership with them that is a financial boon for our professional organization.)


    2. I would add that Jim Wright’s description of the SSR editorial office’s size and procedures sounds very much like an editorial office with the resources for publishing 500 pages a year that is now being asked to publish 1600 pages a year.


  7. I think an important point in l’Affaire Regnerus is the the Editor James Wright DID seek out Dr. Rosenfeld from Stanford, who has experience researching the effects on children who have same gendered parents, Wright sought him out to do peer review and told him he wanted it back within TWO WWEEKS. Rosenfeld agreed to do it, BUT he wanted access to the raw data and Regnerus refused.

    #1 Wright was aware that he should get topic experts to do review and did search them out.

    #2 It is telling that Rosenfeld said he was given two weeks, because if you read the Sherkat Audit and Wrights comments the peer reviewers returned their reviews in 6, 9 & 13 days, meeting Wrights 2 week deadline.

    #3 Wright decided the priorities, obviously Wrights priority was speed over substance. He obviously took any warm body who agreed to do the review, care less if they were experts in sexual minority research or not. This is VERY WRONG.

    The study should be retracted and sent back to peer reviewers who ARE topic experts if Regnerus wants to have it re-published.

    Back to Neal Carens topic which is the Schumm Commentary. I would like even today to have happen what I asked for back in August PRIOR to publication. I would like the ON-Line version of Dr. Schumm’s commentart to have at the top “Commentary NON PEER REVIEWED” and to remove the word “Abstract”. Abstract indicates it is a research paper and is very misleading.

    Can I ask everyone reading here to simply send Dr. Wright an e-mail and ask for these changes to the Schumm Commentary? Sending an e-mail and requesting this change is not that taxing on your time. I think we can all agree that the Schumm “Commentary” needs clarification to the readers.

    I will add this note as to WHY I am asking all of you to ask Wright to make the changes to the Schumm Commentary. While all of you are off doing your Sociology work I am doing different work. I listen or watch all of the Legislative Hearings on Civil Marriage for Sexual Minorities (even in New Zealand and Tsmainia I listened to those hearing). In the Rhode Island Hearing a Psychiatrist from Rhode Island testified and told to the Senators (or it might have been House members) about the Prof Schumm out of Kansas and why is nobody listening to him as he is doing research in this area. This was said right after his comments on Regnerus.

    A Psychiatrist. That Psychiatrist is probably NOT referring to the printed issue of Journal Social Science Research, he is referring to the on line Schumm Commentary which remains published, on line, without any indication that it is Commentary and NOT a Peer Reviewed Paper. This IS being USED as Justification to DENY Sexual Minorities Equal Civil Rights. It was a marathon 12 hours session with no breaks so I am not gonna go back and find it and cue up the link. You can just trust me on this that is what was said in public testimony.

    It is disturbing to me, although I have developed a thicker skin as time wore on, it is disturbing to me that I am derided as merely a gay rights activist as if I should have no voice, as if I don’t have a right to come to you Professional Sociologists and ask for action. Remember the science does not just belong to YOU, it IS out in the Public Domain and affects ALL of us. It has been frustrating to me to essentially be told, “Butt out we are the Pros, we know how to handle this”. You have NOT been handling it, you have not. It is almost ONE YEAR since this was published on June 11, 2012. In fact the reason this is again picking up notice is because of the Supreme Court Hearings, *now* everyone realizes how damaging this erroneous paper IS.

    As much as I appreciate different Sociologists bubbling up the issue again I am leery that the same damned exact thing will happen as when it was first published, Table Talk and nothing else. Although you may wish us gay rights activists to leave you alone on your Sociology Discussion Boards I am afraid to do that, to remain silent, because my fear is that you all will just do the exact same thing as when the study was first published and that is just cheap talk. All talk No Action.

    It is after all only the efforts of the Gay Rights Activists (because you all failed to act) it is only through their efforts that evidence is being revealed. I so so so wish the evidence would have come through your professional organizations but when you all failed to act, Gay Rights Activists did act. At least Gay Rights activists are getting to the bottom of the issue. You may not like their methods but you only have yourself to blame. It was because of your inaction that gay rights activists were forced to step in and examine this issue with real evidence, and most important of all they went out and GOT the evidence.

    The American Sociological Association MUST receive a complaint from all of you and a request for an investigation. As members of the ASA the key players agree to maintain ethical standards. This is the proper body to examine the evidence. It is an EXCUSE to say, “Well we won’t even try because they have no enforcement ability against For Profit Elseveir” That is just an excuse because nobody wants to be the one to make an official complaint. The ASA can send a letter to Elseveir after a complete investigation has concluded, if it is justified.

    The ASA would contact Wright and Wilcox and Regnerus ask them to show up and bring their records. If they refuse to cooperate the ASA can officially sanction them. All I am asking you for, is to file a complaint to the ASA to have them investigate. Instead of Freedom Of Information Act requests which I agree are distasteful, if the ASA investigated there would be none of that. It is the collective “you” in Sociology, your *failure to act* that is forcing the Freedom Of Information Act Requests. There is no recourse at For Profit Elseveir to police themselves, the only appropriate avenue is through your Professional Association, the American Sociological Association. You should all be requesting an investigation by the ASA. Lord knows I have asked many times for the ASA to investigate and my e-mails in this issue are not even acknowledged. I work independently on this issue from my living room sofa. I am thankful to the gay rights activists (however imperfect they are) who ARE investigating because at this point, THEY, and they alone, ARE THE *ONLY* ONES doing any investigating. It should have been different.


  8. “Lord knows I have asked many times for the ASA to investigate and my e-mails in this issue are not even acknowledged”

    Here’s a tip: write shorter e-mails. And blog comments.


    1. 99% of my e-mails or blog comments are at most a couple paragraphs. This is the first time I have commented here and had a lot to share after having followed this now for practically a YEAR without interjecting myself into these Sociology Blogs. Thank you for the suggestion.


  9. Perhaps we should call on publishers to retract Aristotle’s History of Animals for the claim that men have more teeth than women, a demonstrably false claim based on verifiable evidence with an inherently sexist premise? We should cleanse our theory courses of Marx because his predicted worldwide communist revolution never happened. We should probably consider retracting the Feminine Mystique because Friedan claimed to speak for all women when Collins, hooks and others showed very clearly that her conclusions really only applied to white, middle-class women. While we’re at it, we should also ask physicists to retract Newton’s three laws of motion because Einstein convincingly argued they only served as approximations of reality.

    I do not wish to compare Regnerus to great philosophers of history. He is not. He wrote a bad paper. A very, very bad paper. The editor and reviewers made mistakes. The political costs of such a mistake were high and, for that reason, the mistakes of the author, editor and reviewers were more egregious because such controversial findings should be very robust.

    But our colleagues on this blog corrected the scientific record. People knowledgeable of the literature (including those of us only peripherally involved) will now require that anyone citing Regnerus’ paper also cite Andy, Phil, and Neal’s paper. We should not pretend that science is perfect, only that it will correct itself over time and come to the correct conclusion. It seems to me a dangerous project to attempt to scrub research from the record, which is effectively what a retraction aims to do. For this reason, I find Jeremy’s analogy of annulment particularly apt. I think that we should divorce ourselves from Regnerus’s article, not pretend that it never happened.


    1. Last time I checked no one was making policy recommendations or using Aristotle in Supreme Court hearings. Nor was that work claimed to be peer reviewed. Correcting the scientific record among ourselves is not enough if it is still considered legitimate science to non-scientists.


      1. PS – I am not arguing we should expunge the article from the public record, just do a serious review about whether the publication process followed the journal’s protocol (which it seemingly did not) and note that it is retracted before allowing others to view it with a big bold note at the top that it has been retracted.


      2. The implication that I see from your comment is that we should focus on retracting articles when they reveal politically sensitive results. That seems a dangerous precedent to me. We need less politicization of science, not more.

        You will note in my comment that I said that the author, reviewers and editor should have ensured the robustness of the results because the study “found” controversial results that tended against established research. Once the journal (with, as Neal points out above, an underfunded staff) published the result, then the remedy becomes better research to show how they are wrong.


      3. That’s not the implication at all. Certainly these findings were called into question because they were politically relevant and socially sensitive. Now that they have been, there has been lots of indication that the review process did not proceed as it should have and/or were conflicts of interest disclosed, we need to figure out whether it needs to be retracted. That is, was the review process compromised?

        Btw, the same questions (about retraction) are being asked about the Reinhart-Rogoff study, where the error was found out as part of a grad student’s methods assignment, not because of political controversy surrounding the findings. There, however, the review process seems to have proceeded adequately (emphasis on seems) but even still there are questions about retraction.
        cf. http://www.businessinsider.com/herndon-responds-to-reinhart-rogoff-2013-4

        “Once the journal (with, as Neal points out above, an underfunded staff) published the result, then the remedy becomes better research to show how they are wrong.” – this is just not enough, whether the review process was compromised because of it being understaffed or because of political interests (I’m not saying it was either) is not the point, the point is if it’s found to not have gone through proper peer review it should be retracted.


  10. Fine, let’s protect science. What’s our plan to undo the damage done to LGBT people by Regnerus’s study? (A study we seem so deadset on letting stand as sound science.) What do we do to protect marginalized communities from (bad) science?


  11. I agree with Mike about the eventual correction as researchers accumulate more and better data. But that doesn’t seem to be what’s at issue. I get the feeling that I’m eavesdropping on a sociology synod and that besides the sensible and technical talk about rules for retraction, there’s also something like a concern for purity. So maybe we should try to be clear about who this is for and who it will effect.

    Retraction, like excommunication, is about making Us feel good. We want to cleanse ourselves of some impurity. But while that cleansing may make us feel better, will it make any difference outside the walls of the cathedral? I doubt it. The Regnerus study and all the attendant publicity are out there. A barely audible note from a journal saying, “the jury will disregard” isn’t going to change that. Further, I’m skeptical as to whether the Regnerus paper has done any “damage” to LGBT people, though I have seen no evidence on this question. Nor do I think the paper has had any effect on judges, legislatures, or voters. If Regnerus had done the study correctly and concluded that the gay-lesbian variable had no effect, would Scalia change his vote? And if the research had been up to the standards of most publications yet had the same findings, would it change your own political views?

    Maybe there are useful parallels in the Rind Psych. Bull. meta-analysis of 15 years ago. (I’m surprised nobody has mentioned that controversy.) Unpalatable finding, big reaction from the people you’d expect – most political, some methodological, some ad hominem (accusations that the authors had prior, contaminating associations). Some waffling by the journal in the face of much stronger pressure than what the ASA might bring, but no retraction. Further reviews and re-reviews and controversies over the re-reviews. And so on. But if the paper changed anyone’s mind or behavior, I haven’t seen any reports of it.


    1. Except there was no world wide campaign to deny people who were sexually abused as children Civil Rights was there? Did over 200,000 people gather in Paris to march against adults who had been abused as children?


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: