25 thoughts on “the peer review system is broken”

  1. A perspective from a former managing editor (Soc Meth) and student editor/grunt (AJS) – we have ourselves to blame:

    1. Soc journals – rightly or wrongly – bench reject very few papers. Everyone gets to be read.

    2. About half the profession simply doesn’t review papers. Perhaps its different at Mobilization, but that’s my experience. Many people stop reviewing after tenure.

    3. Many reviews that are submitted are clumsy and of low quality, requiring editors to solicit more reviews.

    4. Editors often rely on 3-4 (or more) reviews when 1-2 will do.

    5. People volunteer to be editorial board members, but many editors fail to call these people. Many editorial board members are flakes, which increases the need for outside reviews.

    I bet you probably already knew all of this, but if the issue is ever to be addressed, we have to name and remedy the basic problem: we want everything read, but many of us rarely take the time to read others’ papers. Reciprocity is scarce within our ranks. Sad, but true.

    A grad school adviser once told me it was different before – there was a strong norm for doing quick reviews, which lessens the burden on everyone. But somehow that disappeared and we’re now stuck with no clear way out. If you have any strategies for changing things, I’d love to hear them.

    Like

  2. It seems like an economic problem (my knowledge of economics is basically that of Father Guido Sarducci: supply ana demand). On the demand side, are the editors. How do they select reviewers? How many of those 14,000 people (plus how many non-member sociologists) can they actually link to to make their demand? On the supply side, what are the rewards for reviewing that would bring potential reviewers into the market?

    Like

  3. I was discussing this problem the other day. The key phrase in your letter is “gate keeper.” A select few individuals who are willing to do reviews have disproportionate power in affecting the research agenda. This is dangerous and antithetical to the peer review process.

    Like

  4. I like the idea of bench rejections by editors. I submitted an article to a public health journal and got it rejected in 48 hours. Now that was a good feeling. I think that was on fit rather than quality, because it got good reviews somewhere else. Maybe Soc journals don’t reject on fit, and believe rejecting on quality requires external review. I think Soc editors should reject more often on both grounds: this is not interesting/important enough, and this is not good enough. If you do it quickly it’s much more painless (compare to waiting 9 months for rejection based on a 4-line review).

    Like

  5. I am wondering how this pattern correlates with sub-field. Perhaps the Publications committee is best suited to answer such a question. It would be valuable to know if there are problems generating reviewers in particular fields, or for particular methodological approaches. There could be a lag effect in some fields, as graduate students and young scholars write on topics or with methods about which few mature reviewers are expert. I know I’ve had to work hard to find alternate reviewers for papers on rap music/hip hop culture–both because I don’t have enough time to do all the reviews I am sent, and because I am sensitive to the gatekeeper issue.

    Like

  6. One way to reduce the reviewer load is to stop getting new reviewers on the second round of an R&R. This is not only frustrating from the author’s perspective, who has to please a whole new set of reviewers after making changes to explicitly please the first 3, but also annoying to the new reviewers, who wonder if they’re repeating what has been said before.

    Like

  7. A certain high ranking journal has sought, over the past couple years, to expand the reviewer pool by selecting senior level grad students with publications and professors from lesser known schools in addition to all the regular/typical high prestige reviewers. When this effort was mentioned at the annual meeting, a number of high prestige professors on the editorial board felt an expanded reviewer selection process would lower the quality of reviews and articles published across the board. Of course, a lot of the people complaining about new reviewers are known for turning down requests to review and taking forever to complete a review that is less than helpful.

    It’s true that there are 14,000 ASA members, but are all of them qualified to review?

    I think there are many strong scholars at small schools and colleges who never get a chance to review a paper from a major journal. One reason smaller school scholars are ignored is lack of web presence. Having a CV posted makes it much easier to see if a given reviewer is a good fit.

    Like

  8. Fabio and Trey have a lot of this nailed, and no, Dan, you’re not a crank!

    Some of this starts as a classic freerider problem, and I’ve seen this with a number of friends and colleagues. They want quick and thorough reviews, but they are too busy and important to do them when they are asked. Or, worse yet, they regularly agree to review manuscripts and then they don’t complete the reviews. The reason why Dan and others have stacks of manuscripts sitting on our desks is that we’re on the ever-shrinking “good” list. Now, with centralized manuscript processing, reviewers are often being selected based on their delivery, and the systems make it even easier for editors and the editorial staff to tell who completes their reviews and who does not.

    Even worse, many of our shirker colleagues are socializing junior scholars into freeriders. They tell students and junior colleagues to only agree to review if it is at a top-top journal, and only if it is a manuscript substantively and methodologically identical to their own research. I blew a fuse editing a couple of special issues of modest journal when I kept getting lame ass refusals from junior (and not so junior) people–with publications in the field–saying things like “I study Korean Christians, I can’t evaluate a manuscript on Chinese Christians.” Or, “I’m giving a talk at Harvard next month, I can’t possibly review a manuscript.” And, of course, if I ever agree to edit anything again, I won’t even try to call on those people. What’s the point?

    I think large, diverse, rotating, and WORKING editorial boards are the answer. If you’re on the board, you’re going to be beaten to death. Then, in three years, you’re free (and a new set of working gatekeepers with different perspectives can fill the bill).

    Like

  9. I’ve complained before about this so you know I’m on Dan’s side. And since you raised the issue, here is my rant. In the early 1990s I was doing five reviews a MONTH, I was on EVERYBODY’s “good” list. I had reviewed prior to publication over half of the major articles in my area. Several times I have gone through a spell of saying “no more for a year” while on sabbatical or being chair which cut it back, and I just HAVE to say no to a lot of the requests, because I get several a month every month.

    Are we willing to sanction negative incentives? How about each journal publishing a list of tenured people who publish in that journal who refuse to review for it?

    And to the editors of “lower tier” journals, I say this: you folks are part of the problem. You want your reviewer pool to be the same pool as ASR/AJS and you want the same standards applied. You get offended if a reviewer says, “this is good enough for your journal.” I get asked to do reviews for a lot of journals I have not even heard of, much less ever thought to publish in. And for journals that I have heard of that are well understood as “good second tier” journals. Part of the annoyance is having to tiptoe around the editor’s ego and trying to write a view that is honest about the work’s quality and importance while still taking into account the character of the journal. No journal should publish stuff that is wrong, but articles that are right do vary in their importance.

    The fact is that reviews vary in quality, not just in whether they get done at all. “Good” reviewers are people who actually read the articles, understand the methods of the articles they are reviewing, are relatively free of biases and agendas in their reviewing (apart from a belief in the necessity of sound argumentation and method). Even good reviewers can make mistakes, which is why editors like to see multiple reviews. Less experienced people are often not good reviewers, either because they are too critical and cannot separate the important from the unimportant in their reviews, or because they don’t have the experience and knowledge to read well outside their own specialty. Teams of graduate students can often make good reviewers as they aggregate their knowledge, as can graduate students supervised by a professor.

    I think Sherkat is probably right about editorial boards being the best approach.

    Like

  10. PS To re-state a point I made the last time we went around on this issue, I estimate that a tenured person owes at least 6 and probably 12 reviews for every article submitted. The logic is that well over half of articles submitted are by graduate students or unpublished new PhDs, that no journal in any tier can rely for its decision only on the reviews of unpublished novices, that unpublished people have a substantially higher ratio of submissions to publications than published people, and that each submission will occupy the work of at least 3 reviewers.

    Tell me where my reasoning is wrong here.

    Like

  11. I think you have established a social justice rule, now I suppose we should try to make it a norm? I’ll have to recheck my Willy Jasso papers.

    OW’s got it. I’m strongly agree regarding 2nd/3rd tier journals. I just received a reviewers’ packet from a decision on a paper I reviewed in two rounds. I wanted a conditional accept on the first round, and it received an R and R. The second reviewer from round 1 was also very positive about the paper (clear R and R), but was not the second reviewer used on round 2. My review of the revised paper was one line long. “This is a solid contribution to the literature in X and should be published” A third reviewer was invoked on round 2 (as warned against by Brayden), and that review was overly nitpicking for a journal at this tier. So, the author(s) got a second revise and resubmit, “permit a revision”, even. I’m on the editorial board of this journal, and I published a paper on a very similar topic last year. So, now more reviewers will have to be bothered before this nice-but-not-spectacular cocntribution appears in a modest journal?

    Like

  12. So as an end-of-graduate school/new PhD. I have been told (and I agree) that I should review everything that comes my way…no matter how far from my area (luckily nothing too crazy yet). I thought I was reviewing a lot at 2-3 reviews/month. Since I submitted 6 articles for review last year, that would suggest that I need to review 36-72 articles per year. At best, I am just under that range…I think I am starting to get light-headed! OW suggested that the 6-12 applies to tenured people…so what is my level?

    Like

  13. Bandeiras: congratulations on your productivity! I envy you. And also on your public service, as it sounds like you are already doing 24-36 reviews a year, if I correctly understand your statement. This seems very good for a new PhD. My calculation was for a tenured person, making the assumption that young people like you would be doing fewer reviews than submissions. If you are getting asked to review that many things that young, you must be a “good” reviewer.

    Like

  14. I’d co-sign your letter.

    First, I’m reviewing too many papers. But I’m nothing. When Chuck Tilly died our department received notes from journal editors who said that Chuck often did the anonymous (thankless) work of reviewing papers at an astonishing rate – sometimes more than 25 reviews per year (just one journal!). So maybe part of the problem is we’re all picking up Chuck’s slack. He’s a case of small groups of people doing the lion’s share of the work, though an extreme one.

    Second, I think the “blind” part of the peer review system is often a farce. It’s a farce because of either I know who wrote the paper as soon as I read it (I’ve seen it a conference, talked to others about the project, etc.) – OR, the editors manage to reveal it by not masking the identity (through accidents like sending the complete reply letter on an R&R, so I know who it is. I will send notes to editors telling them this. “I know who wrote this. It’s X. I am not interested in either promoting or destroying this person’s career. But you should know that I know (or REALLY think I know).” No editor has ever said, “Never mind. We’ll find another reviewer.” I guess that’s because if I were interested in either promotion or destruction, I would never send that note. All I’m saying is that the benefits of “blind” are often over-played insofar as it’s not uncommon that it’s a fiction.

    One final note on editorial boards: Politics & Society uses such a board and I’ve heard complaints about nepotism with that journal. Then again, I’ve heard AJS critiqued in the same way (called, “a fiefdom” and noted that folks get published there and often not elsewhere). As far as I can tell, neither of these are really true. But I wonder what unintended consequences such boards might have.

    Then again, I like the idea of public, rather than anonymous gatekeepers. It’s easier to hold folks to account.

    Like

  15. This is a very interesting exchange, especially in the light of the new policy recently announced in EPD. They have added a new condition to submission of a paper: “that was surely implicit, but we have now made explicit:
    – that you agree to act as a referee for other submissions to the journal, bearing in mind that each submission generates the need for three reports.” (p. 953)

    Their logic seems to be that since each article submitted generates three reviews, each author submitting an article should do three reviews. But I definitely see the relevance of distinguishing between tenured/non-tenured or well-published/less-published reviewers.

    Elden, Stuart, 2008, The exchange economy of peer review, Environment and Planning D: Society and Space 26(6) 951 – 953.

    There’s a lot to be said for networking here. If I am approached to review on topics that really are too far from what I do, I try to provide a list of other people (who the editors might not know) who would be better placed to review.

    Like

  16. You know, I would be happy to review more papers but I’ve only been asked to review a paper ONCE for a “second tier” journal. I have a Ph.D. but a non-academic job and relatively few (but not zero) publications. Are people like me considered unqualified to review? (I am an active member in my ASA section and this still doesn’t translate to review requests, though it does to other kinds of committee work.)

    Like

  17. I wonder if some of this has to do with professional socialization in graduate school? When I was in grad school, we were told early and often that reviewing for journals is a professional obligation and the only legitimate reason to decline is if the article is so far out of your area that you couldn’t review it. For a generalist journal, such a situation should be quite rare. I wonder if some schools don’t teach people how to review? We got a full semester course focused mostly on just that topic! I have reviewed probably 8-10 papers and have not found it to be difficult at all, even when the papers were pretty far from my sub-field (although clearly I am not dealing with the volume that most of you are!).

    Like

  18. As a recent PhD/junior faculty member, I actually enjoy reviewing papers, and wonder why I am not sent more of them. The strange thing is that I have rarely been asked to review papers for journals where I have published. Instead, I seem to be contacted primarily to review papers that are extremely relevant to my own research – sometimes for non-sociology journals (geography, anthropology). I guess I am selected because of my very specific knowledge of particular topics. Anyone know why a (second tier) journal would not solicit reviews from its own authors? My best guess is that they are seeking first-tier reviewers.

    Like

  19. @mantruk – I’m too early in the process to speak to the rest of this thread, but I can definitely say that reviewing is not a skill I’ve been formally taught in any class. One of the workshops here requires graduate students to write anonymous reviews of pre-circulated papers, which has been very helpful practice, but we never receive feedback on those reviews. So… I doubt I will have any formal or even thorough informal training in how to review a paper before the first request for review gets sent to my inbox.

    Like

  20. This is rather logical, you know. People get work done in the summer and over winter break, so I imagine there is a glut of things that need review in Sept-Oct and Jan-March.

    Like

  21. Since I posted my comment 2 days ago, I have received 2 requests for reviews. And as much as I would like to blame that on Scatterplot, I am fairly certain my name makes me anonymous. So, it must just be karma! (Or OW’s logical explanation.)

    Like

  22. It’s just one person’s data, but I’ve never perceived that this legendary glut of submissions after summer and winter breaks is real. We all want to finish things up and get them out then, but things always take longer than we plan and so it doesn’t really end up working out that way.

    I went on my hard drive and recorded the month for every review I’ve ever written (for journals–obviously grants and other kinds of things have different period logics). If OW is right, I ought to see more than 5/12 or 41.7 percent of reviews in the five months she mentions (Sept, Oct, Jan, Feb, March). Again, it’s just me and I might be weird for some reason, but only 40.8% of my reviews were written in those five months. My heaviest months were May and July.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.