blind no more!

Not that anyone in sociology has ever used google to identify the author(s) of a paper they’re reviewing, but: The American Economic Association has decided to end double-blind reviewing. Going forward, you will know the authors of the paper you are reviewing. The rationale:

Upon a joint recommendation of the editors of the American Economic Review and the four American Economic Journals, the Executive Committee has voted to drop the “double-blind” refereeing process for all journals of the American Economic Association. The change to “single-blind” refereeing will become effective on July 1, 2011. Easy access to search engines increasingly limits the effectiveness of the double-blind process in maintaining anonymity. Further, it increases the administrative cost of the journals and makes it harder for referees to identify an author’s potential conflicts of interest arising, for example, from consulting.

Should we?

34 Comments

  1. Posted May 27, 2011 at 9:07 am | Permalink

    Should we: I don’t know. I can think of good reasons for both sides. But I think this definitely makes sense for econ where working papers are so regularly put online. We don’t have such an ethic. Though it’s still easy enough to guess who wrote a paper.

    I would be interested to see what this does to the proportion of papers published by already established people. Natural experiment, baby!

    Like

  2. Posted May 27, 2011 at 9:34 am | Permalink

    BTW, is book reviewing ever double-blind? (For graduate students who might not know, federal grant reviews are certainly not double-blind and I don’t think many other kinds of grants are. The TESS project that I run does use double-blind reviewing.)

    Like

    • Posted May 27, 2011 at 9:59 am | Permalink

      Do you know if anyone has looked into differences between high-status federal grant recipients and publications in high-status journal (by status of the institution of the applicant)? It would be interesting to see if blind vs. non-blind processes are more likely to reward existing status.

      I can’t image that it would be possible to have blind book reviewing… How would you blindly read a book? Perhaps as a PDF… but unlikely.

      Like

      • Posted May 27, 2011 at 12:40 pm | Permalink

        Books are not double blind reviewed. Also, journals in biological fields are often single blind.

        Double blind *might* be better, but there’s precious little evidence either way. I applaud the AEA for at least recognizing that the anonymous peer review era is now ended.

        Like

      • mike3550
        Posted May 27, 2011 at 3:31 pm | Permalink

        Not just pure biological fields as Fabio notes below, but also in many public health journals the process is not double-blind.

        Like

  3. Posted May 27, 2011 at 9:50 am | Permalink

    I fear that single-blind would disadvantage grad students (as googling names already does), at a time when they are already facing an uphill battle.

    Like

    • Posted May 27, 2011 at 9:57 am | Permalink

      I wonder about this. I might be more willing to give a shot to a grad student than, say, someone who has published 100 articles already. I’m not sure how this would work. But it seems to me that there are reasons to imagine that single-blind could go either way.

      Like

      • Posted May 27, 2011 at 12:01 pm | Permalink

        I’m with Tina on this and it’s because I think egalitarian sympathies will always be swamped by awe of status.

        I recently reviewed a paper and I and the other reviewers all had severe problems with X. The editors rejected the paper but avoided the issue of X, instead suggesting they were doing so for issues related to Y (an area where I thought the paper was actually strong). I told my wife this puzzling story and she recognized the paper from a conference presentation as being the work of a world-renowned expert on X.

        I think that if I knew who the author was I wouldn’t have been confident enough to point out the problems with X, even though having come to my assessment blind and only in retrospect learning the author’s identity, I am absolutely confident that there were crippling problems with that aspect of the paper.

        (e-mail me if you want the non-redacted version of this story).

        Like

  4. Posted May 27, 2011 at 10:08 am | Permalink

    As an author, I was unnerved when I realized that Social Science Research was single blind (which wasn’t until I got reviews back).

    However, I’ve appreciated it as a reviewer. I would have known the identity of most of the authors anyhow from presentations and professional networks. Having the identity out there, in many ways, has made me more aware of my potential biases. If anything, like Shamus suggests, I’ve also found myself rooting for the underdog.

    Like

  5. morgansl
    Posted May 27, 2011 at 11:07 am | Permalink

    I think we should be enthusiastic about anything that speeds up the review process. My (very tentative) guess is that such a shift would speed up the review process for established authors and slow it down for graduate student authors. That’s probably a net loss to the discipline, since established people can afford to wait and probably have lots in the hopper already.

    In any case, with google already revealing lots of identities, maybe all of these effects would be quite small anyway.

    Like

  6. Posted May 27, 2011 at 11:12 am | Permalink

    We should keep double blind reviewing. Guessing at authorship isn’t the same as knowing and makes it more likely that we’ll put our biases to the side. As someone outside of academia (who still occasionally reviews papers), I might not know the authors anyway. But for those who do, I the double blind review is necessary to make sure that we don’t appear to have bias in favor of the work of our friends. Not that we would actually have this bias, but as the government ethics rules remind us, we need to avoid the appearance of impropriety as well as actual impropriety.

    I also wish we could do our federal grant and proposal reviewing blind. My office often gets accused of playing favorites and I think a blind reviewing process might be enlightening. Unfortunately, we are required to rate proposals on “past performance,” which makes a blind review impossible.

    Like

  7. Posted May 27, 2011 at 11:24 am | Permalink

    I have Google Analytics on my personal/academic home page, so when people Google the exact title of my paper currently under review, I know the city where the search came from. In some cases (especially with college towns), it’s a pretty strong clue regarding who reviewed your paper.

    I suppose I could take the paper off of the website, or change the title, but I’d rather not do that while on the job market. I could also uninstall Google Analytics (in the case of knowing one’s reviewers it’s “TMI”, in my opinion), but that wouldn’t change the fact that reviewers Google papers and I am identifiable.

    With today’s information technology, double-blind review might not be more than 0.5-blind review!

    Like

  8. Posted May 27, 2011 at 12:23 pm | Permalink

    I think there is some merit to double-blind even though it isn’t really blind. The most important part is having the reviewer be unidentified.

    Minus to identifying author IMO is the “halo” effect for a high-status author. My impression as a reviewer is that there substantially less risk of decline in the evaluation of a good paper from a low status person than the risk of increase in evaluation of a mediocre paper from a high status person. I do think reviewers would be less likely to be “helpful” in the case of a flawed paper if they know the writer is high status. If somebody with tenure sends out a bad paper, you’d be more inclined to just reject it with insulting but not helpful comments.

    Even if the authorship can be figured out, the pretense of anonymity seems to me to promote better behavior by reviewers in the writing of the reviewers.

    The only negative I can see to double-blind is hidinggrad’s point that you can more easily identify a reviewer if they’ve googled you. But then if people know to set up google analytics on their sites, this can be a deterrent to the practice.

    About grant proposals: part of what you are supposed to evaluate is the person’s capacity to do the research based on their past performance. Predicting the future is different from evaluating a written product.

    Like

  9. Posted May 27, 2011 at 12:28 pm | Permalink

    Blind please! NSF reviewing is very prone to what I call “trust me bias” where senior scholars are not called out for stuff that will kill a proposal from a neophyte. And speeding up reviews for seniors will indeed slow it down for juniors (or else people will be loath to even review a senior who can be annoyed. These don’t balance out so much as confound each other. Guesses about who the author is can be wrong and I like that. A friend once said she had “reviewed my recent ASR submission and leaned over backward to be kind to it” for which I was obviously supposed to be grateful. But when I said I was surprised since my only recent submission had been a quant paper and she was a very qual researcher” she was nonplussed and admitted she might have been mistaken. I don’t think I would lean over backwards to do favors (and know I would not brag if it did)but I’m pretty sure that most of the leaning over backwards would accommodate seniors not juniors. And while I can only speak for myself, I’m happy tore IWW blind and would not een think of googling to find out an author, even if I were sure I could.

    Like

  10. Posted May 27, 2011 at 12:37 pm | Permalink

    As a related tangent, exercise caution when guessing who your reviewers might be. Typically we’re probably wrong when we think we know the identity of the reviewer. I’ve experienced two occasions where someone thought they’d guessed I was a reviewer. In one instance, a person thanked me for the positive review, to which my response was, “You’re welcome but it wasn’t me.” In another instance an anonymous reviewer on a paper said I should do a better job of concealing my identity when writing a review, to which my response was, “Someone clearly is doing a good job of impersonating me because I wasn’t the reviewer.” I can only wonder how many times people have assumed I was a reviewer when in fact I wasn’t.

    My point is that we’re probably much less accurate at guessing who reviewers are than we think we are. There’s no sense having bad feelings toward someone for giving us a bad review when there is a low likelihood that we know who that person really is. Of course, if you want to incorrectly attribute positive feelings to me for writing an awesome review, please feel free to do so!

    Like

    • dantae
      Posted May 27, 2011 at 1:06 pm | Permalink

      I have heard the exact same thing from people at two different journals: It is apparently common for people to angrily email editor saying this reviewer is a particular named person who is biased against me. Editor checks and it is not that person.

      On a further tangent, when I act as a reviewer, I usually sign my reviews. Perhaps this is unwise as a junior person, but I am proud of my reviews and stand by them. I also think it probably works to my advantage because the other reviewers generally see my name.

      Like

      • Posted May 27, 2011 at 1:18 pm | Permalink

        I think it is bad to sign reviews. Too much risk of quid pro quo. I’m proud of my reviews, too, I work hard on them, which is why I’m now having to tell journal editors to leave me alone. But anonymity of reviewers is the only way to keep cronyism out of the system. Even if it isn’t a problem now when you are junior person, it will be when you are senior.

        Like

      • Posted May 27, 2011 at 1:20 pm | Permalink

        PS that’s also why I don’t think it is such a good idea to go around telling people that you gave their article a positive review. It’s just asking for payback IMO.

        Like

    • Posted May 27, 2011 at 1:13 pm | Permalink

      That’s my fault. To keep it interesting I try to write all my peer reviews in your writing style and dropping references to your work. It’s practice for when I write a series of detective novels in which a young b-school professor solves murders in and around Evanston.

      Like

      • Posted May 27, 2011 at 1:31 pm | Permalink

        Nicely done. I’ve decided to counterstrike by signing all of my reviews, G. Rossman.

        Like

      • cperchesk
        Posted May 27, 2011 at 1:32 pm | Permalink

        If you need to do field work for the novel, you’re welcome to visit!

        Like

  11. Posted May 27, 2011 at 12:58 pm | Permalink

    I’m with the “keep double blind” crowd. Of course I understand the google effect, and I have been guilty of engaging google to assuage my curiosity. But there’s a difference between submitting a critical review when you know who the author is but they don’t know you know, and submitting one when they know you’ve been told who they are.

    I also think it’s a matter of irrelevant information — what benefit is served by the name of the author being released to the reviewer? Either way it ought not change the content of the review.

    Like

  12. Posted May 27, 2011 at 6:12 pm | Permalink

    I remember as a grad student being told that the reaction you wanted to provoke in reviewers (when sending in papers as a student or young faculty member) was to have them sit there, impressed with the paper and wondering “Who *is* this?”—rather than have them thinking, “Meh, some grad student”. So my gut reaction is that switching to single-blind amplifies Matthew effects, even if often the anonymity of double-blindedness is a sham. So I’d concur with most of what OW said.

    If it ends in Sociology as well as Econ, I suppose I’ll just start unilaterally adding widely-praised, very smart people as co-authors.

    Like

    • Posted May 27, 2011 at 10:57 pm | Permalink

      You’re welcome to add my name to one of your papers any time.

      Like

  13. Posted May 28, 2011 at 1:23 pm | Permalink

    Sociology is such a small world I’d rather not know who I’m reviewing if at all possible. I had the experience with a single blind journal where it turned out that the person I was reviewing had been interviewed for the same position I ended up getting (he got a much more lucrative one elsewhere). I would rather have not known who he/she was even though it was a very good article. I also really don’t want to know who slams me with a nasty review. Can you imagine if we all knew that how different the ASA’s would be every year? :-)

    Like

    • Posted May 28, 2011 at 2:17 pm | Permalink

      I assume we’re only talking about single-blind in any case, i.e., the reviewer would know who the author is but the author wouldn’t know who the reviewer is.

      Like

      • Posted May 28, 2011 at 3:25 pm | Permalink

        Yes of course. I was just making reference to mentions here about guessing who your reviewers are. I don’t want to know and I think it is best that we all don’t know. Also as its mentioned here, our guesses can frequently be wrong. The ASA mention was just a joke.

        Like

  14. Posted May 29, 2011 at 5:41 pm | Permalink

    Most people’s reactions to this, and predictions about what will happen, appear rooted in theory or in personal experience/anecdote. There is some evidence. Rebecca Blank did a very good randomized study of this in economics, published in 1991 (http://www.jstor.org/stable/2006906). Main findings (to this reader): acceptance rates go up under single-blind reviewing and reviews are less critical; men and women are affected equally; and people at high-status and low-status university are not affected by the change, but authors in the middle of the status distribution have a higher acceptance under single-blind reviewing.

    The status effect is fascinating, and does not conform to simple priors like “reviewers will defer to high status actors.” It is probably true that knowing that the author is high status induces some deferential behavior, but other processes might be activated as well and how these balance out is unclear.

    Another interesting tidbit from that study: in the late 1980s, when that study was conducted, almost half of reviewers in the double-blind condition correctly guessed the author. This was, mind you, pre-Google. The information value of revealing the author has to be really small today.

    Of course, it is possible that sociology is totally different from economics in this regard. Seems unlikely, but possible.

    Like

    • krippendorf
      Posted May 30, 2011 at 7:42 am | Permalink

      I wouldn’t say “totally different,” but it seems plausible that status effects — whatever they are — would be amplified in soc relative to econ, at least if any of the stereotypes about differences between the disciplines are true: less agreement (in soc) about what constitutes good work, less adherence to a single orienting paradigm, and weaker / fuzzier boundaries between subfields, which makes in more likely that a paper will draw one or more reviewers who are only minimally familiar with the area.

      Like

  15. Posted May 29, 2011 at 9:21 pm | Permalink

    Thanks for digging the cite up, disgruntled.

    I understand people’s status concerns and remember seeing Robb Willer present a paper at the ASA meetings years ago that explored the link between status and text evaluations.

    The comments above provide compelling reasons to keep things double-blind. However, they also left me with lingering questions. For example, I wonder 1) if we’re not also influenced if we think we know who wrote the paper (in the same way that those who think that they know who wrote their reviews are) and the ramifications of such assumptions, 2) if in a situation like Gabriel brings up, reviewers familiar with X still wouldn’t have criticized the author about their treatment of the subject (does something need to be unintelligible for us to fall victim to what Willer describes in the paper above?), and 3) if contextual information about the author couldn’t actually improve reviews and recommendations (particularly the question about the chances this manuscript could be successfully revised, or the recommendations reviewers make and the tone/clarity with which they write them).

    Like

    • Posted May 30, 2011 at 12:51 am | Permalink

      I agree that it is hard to tell which way the effects will go. There is of course lots of work, including Willer’s, to suggest that Matthew Effects matter in scientific evaluations. But a lot of these studies are about establishing existence – they do not give much insight into the magnitudes of the effects, and how they balance out with other effects. As sociologists we may over-estimate the impact of status in distorting the outcomes. (One problem in applying Willer’s study is that it was undergrads doing the assessments. If the discipline’s reviewers are no better at checking their biases than undergrads, we should worry.)

      So the questions are what are the costs of double-blind? (Leave the administrative costs aside, even though the AEA invokes them, since these seem trivial.) Several commenters have told stories about people mistakenly thinking that they had reviewed their papers. But of course people only tell us that when they think they have been nice to us; we don’t hear about the times when we thought the author was X and applied a higher standard.

      It is entirely plausible to me that in these cases we are more influenced by an identity if we think we know who it is, because we are not forced to counter-balance our rage at their perfidy with the image of the real person. And it is entirely plausible to me that an availability bias creeps in here — when we start to imagine “who could this possibly be” the names that come to mind are naturally those of high status people. So we think of Prof Z at Chicago, and not of Prof Z’s student from 10 years ago, who is toiling away at a middle-status school. Again, that may work to the advantage of Z’s student, but it may not. The findings in the Blank study suggest that double-blind reviewing works *against* Z’s student, and is net neutral for Z. It’s not obvious that that is a virtue of double-blind reviewing.

      Like

  16. Posted May 31, 2011 at 3:39 pm | Permalink

    As a follow-up and as evidence (of a sort) of the point I was making, consider this from Mario Small, in the post on orgtheory on what makes a good review (http://orgtheory.wordpress.com/2011/05/31/the-editors-speak-what-makes-a-good-review/)

    “In a nontrivial number of cases, cranky evaluations are produced by reviewers who purport to know who the author is (and the targets of these guesses are rarely graduate students). By contrast, when reviewers assume that the author is a graduate student (they often tell us, in the comments to the editors), they inevitably come across as more helpful, objective, and—frankly—rational.”

    This would suggest that beliefs about status do not always redound to the benefit of high-status actors.

    Like

  17. Posted June 1, 2011 at 6:02 pm | Permalink

    I know I’m taking this back to personal experience but when I know who the author is — esp. with an NSF proposal — I evaluate with greater expectations for higher-status actors. That being said, anything, no matter from whom, I evaluate for theoretically and methodologically soundness (regardless of the theory or method), and of course contribution to the existing research. But still I expect more sophistication from the higher-status actors (I’m assuming this is about prestige of university as well as senior level). Ok, but here’s where things might get a bit messy. I’ve been on NSF panels for postdoc awards where some very high status actors are writing letters of support for their students. I have been utterly shocked at how bad some of the proposals from the high status actor’s students have been. I’m mean…like I try very hard to be a very fair and thorough reviewer but this stuff was just plain bad research. So the net affect on me is that I get angry that such a high prestige actor would actually let something like that go through.

    Ok, Yes, I am an N of one here so perhaps others have had differing experiences. But part of my very non-scientific account here supports disgrunted.

    Like

  18. Michael Bishop
    Posted June 23, 2011 at 4:19 pm | Permalink

    Good comments here. Want a good economist’s take? go here: http://worthwhile.typepad.com/worthwhile_canadian_initi/2011/06/adverse-selection-and-single-blind-review.html

    Like

Follow

Get every new post delivered to your Inbox.

Join 1,110 other followers

%d bloggers like this: