blind item

A couple years ago, I reviewed a paper for a highly reputable journal (note: not ASR or AJS). The paper had substantial merit, but did an extra analysis that I didn’t think made sense. (Worse, really, I thought it was outright misleading.) I explained why in my review. The paper was given an R&R in which the authors were instructed to answer various reviewer criticisms. The paper was re-submitted, and, in the revision, the questionable analysis was removed. I recommended acceptance, and, as it turned out, I was the only reviewer on the second round. The paper was published.

Recently, I was reminded of the paper and looked up the published version. I saw that the authors had put the questionable analysis back in the paper.

Does this happen often? Should I be bothered? Annoyed? Upset? Angry? Is there anything I should do, beyond the catharsis of posting this?

Obviously, there are many complaints about the review processes and how reviewers can make authors do things that the authors don’t think are right. I’m pretty confident in my views about this analysis, but the authors may well disagree (after all, they were the ones who did it in the first place). So, I can empathize with a certain degree of “Screw Reviewer C!” thinking. But, well, as Reviewer C, I’m feeling a little defrauded.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

10 thoughts on “blind item”

  1. If you truly feel the analysis is misleading, I would probably contact the editor and explain the issue, reminding him/her why the analysis was misleading, and suggest they allow you to write a short comment. But, as you know, that can snowball into more than you bargained for. So, the initial question I think should be, is it misleading enough and the possible worse case scenario fallout worth it?

    Like

    1. That’s a good way of putting it. *I* feel misled, but I don’t think it’s misleading enough in terms of the science takeaway, and I don’t want to deal with any fallout.

      Like

  2. I do think it would be interesting to know, though, whether the author of the piece was able to persuade the editor that she/he was right and got permission to put it back into the final published draft—or whether the author snuck it back in there in the final stage without getting permission from the editor.

    Like

  3. I’ve only seen things go in the other direction. Additional stuff got put in to satisfy the reviewers. Then, the Editor agrees with the author that the paper has become bloated, and the Editor is happy to let the author do whatever excision will take up fewer pages in the journal. I generally think this is fine.

    What you have experienced is more interesting! I suppose you really need to know the details to take a position.

    Overall, I am coming to a general orientation, which has been building for five years or so, and which is reflected in the way in which we create and run SocSci. After a threshold of quality is passed for the core contribution of a paper, reviewers and Editors should mostly get out of the way. It is then up to the author to bring it home and show what they can do with the paper. If people want to tack on misleading, weak bits to their paper, people who can detect so will think less of them for having done it. The only downside, of course, is that some people will be misled. In the long-run, though, those who are misled probably aren’t the people that are pushing science forward. So here is my emerging orientation: Rather than journals being a flawed quality screen, they should enable people to demonstrate how good or not good their work is. In this case, someone has gotten the opportunity to demonstrate that they are not as good as you would like them to be.

    Like

    1. Thanks. A scenario that would be certainly fine to me would be where the authors write the paper, the reviewer gives a recommendation, and then the editor and author work out what should be done in the final paper. I might be annoyed that they chose not to listen to me, but there wouldn’t be any warrant for being too annoyed, much less feeling wronged. It’s their paper.

      What bothers me here of course is that the editors asked me to volunteer my time twice, and the second time explicitly to evaluation how the paper did with my revisions. I took the time to explain to them why this part should be removed. And they sent back a version in which it was removed, which meant they didn’t have to explain what was wrong about me saying they should take it out. I tell the editor that the problems I was concerned about where addressed.

      Only then it shows up back in the published paper. So, what was the point of the authors taking it out in the version they sent me? Or, what was the point of the editors asking me to read it a second time? It makes me feel like I volunteered my time only to be deceived.

      (It would be even weirder under Rory’s scenario where they appealed to the editor to put the problematic analysis put back in. Couldn’t they have just done this at the start, and left me under the loop? Instead, it’s like there was a special version of this paper that was created just for me to read for a second time.)

      Like

      1. I am not really arguing against your annoyance! But I think in 2014 I would be less annoyed than if this had happened to me in 2004. The key thing is that we continue to upgrade the training we give to graduate students so that more people can sort the good from the bad.

        Like

  4. It would definitely annoy me (as a reviewer) and I would want to know if the editors were aware of it. In the journal I edit, I would catch it in the copyediting process (if someone snuck something back in like that). But with some larger journals, it may be the case that the editors aren’t really paying much attention to the final draft that is submitted for copyediting. And as the editor of the journal, I would want to know if somebody did that since the reputation of the journal–not just of the author–is at stake.

    Like

  5. It would annoy me mightily as a reviewer, and I think you are well within your rights to ask the editors if the author “snuck it back it” or whether they approved the re-insertion. Either is still annoying but then you know whom to be most annoyed at.

    Like

    1. I would be annoyed too. But it doesn’t violate your rights as a reviewer (which are, let’s just say, limited). What it does do is mislead the editor, who probably doesn’t even know it happened. I would get it off your chest by telling the editor and let them do what they want with the info.

      Like

      1. Yeah. If it was a paper that just came out or a point that was truly decisive to the merits of the paper, I might write the editor. But I think I’ll just let it go. Was very useful to talk this out, though, so thanks.

        It could be an interesting topic for another day per Phil and other comments about what exactly the rights of a reviewer are, or maybe not “rights” per se, but under what circumstances a reviewer can feel justifiable umbrage. I agree they are very limited, especially as I share the basic vision of Steve and others about getting the Editor and reviewers more out of the way.

        Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s