I freaked out recently when, after reviewing an article, I received a packet of FIVE (5!!!) reviews on the same article. I chewed out the editors for wasting my time and told them I would never review for their journal again. After an exchange (in which I got a little less testy), I told them I’d post my concerns to scatterplot and open a discussion on the topic. Although five was over the top and freaked me out, it has become pretty common now for me as a reviewer to get a packet with four reviews. No wonder we regular reviewers are feeling under the gun. The old calculation of two or even three reviews per article has gone by the wayside. The pressure for fast turnaround and the high turn-down or non-response rate among potential reviewers has led editors to send out articles to extra reviewers in the hopes of ending up with at least the minimum two or three.
But this is a death spiral. As a frequently-sought reviewer I get at least four requests a month, sometimes as many as eight, and I used to get more before I got so crabby. When I was young and eager, I was reviewing an article a week [and thus, by the way, having a huge influence on my specialty area], and I know some people who are keeping that pace. But at some point you burn out and say “no more.” I, like all other frequently-sought reviewers I know, turn down outright the requests from journals I don’t know for articles that sound boring, and then save up the other requests and once a month pick which articles I want to review. So the interesting-sounding articles from good journals get too many reviewers, while the boring-sounding articles from no-name journals get none. If journal editors respond to the non-response by reviewers to boring-sounding articles by sending out even more reviewer requests per article, our mailboxes will be flooded even more and the non-response rate and delayed-response rate by reviewers will go up even more. Senior scholars are asked to review six to eight (or more?) articles per month. You have to say no to most of the requests.
And then we have the totally out of hand R&R problem. I think it is completely immoral to send an R&R to ANY new reviewers. I know a young scholar with a perfectly good paper who is now on the 4th (!!!!) iteration of an R&R from ASR. Not because she has not satisfied the original reviewers, but because the editors keep sending each revision to a new set of reviewers in addition to the original reviewers and, of course, the new reviewers have a different perspective and a new set of suggestions for the paper, some of which cover ground that was gone over in one or more of the previous revisions. Not to mention the problem that R&R memos are now longer than the original articles!! We are no longer a discipline of article publishing, we are turning into a discipline of R&R memo-writing.
Something has to change. Senior scholars burn out and get reputations for being difficult, possibly because editors don’t know how many other people are asking them to do things. Junior scholars would want to review and wonder why nobody is asking them, and other junior scholars think they are being tapped a lot because they are getting four requests a year. Article-submitters (disproportionately junior scholars) whine and complain about slow turn-arounds, and imagine — what? I guess I don’t know what they imagine? Do they even understand what is happening on the reviewer side of the equation? I think some of the more clueless imagine that reviewers are just queuing up to write negative reviews about them and it is all the editors’ fault for not organizing things better.
My purpose in posting is to open the discussion. I think what is needed are some ground rules that would help the senior scholar problem. (1) Reviewer time is a scarce resource. Treat it as such. Do not waste people’s time. (2) No article is ever sent to more than three reviewers. Better is to send to two and ask for a third if there is a split vote. (3) If a reviewer fails to respond in a timely fashion, they get an email: please respond or we will send the article to someone else. (4) If an editor has three reviews, they immediately send a notice to anyone else they asked for a review saying “we have enough now” or, if you insist, “we have three reviews but they are mixed, and your opinion would help.” (5) If you get two reviews and the situation is obvious, tell anyone else you asked for a review “never mind.” (6) An R&R is sent back to the original reviewers and to NOBODY ELSE unless there is some very specific issue and the paper author is told at the time what the issue is and the category of additional reviewer who will be solicited. (7) Author angst about turn-around time is dealt with not by sending articles out to eight possible reviewers (!!!!) but by keeping authors informed of their status. Telling an author that they are having a hard time getting reviewers lets them know what is going on. (8) Tell reviewers you want a response to the “will you review?” email within two weeks and cancel the invitation if they do not respond within one week to the follow up to the initial request. Leaving the requests open just encourages the kind of gaming I described and increases the risk of wasting reviewers’ time with too many reviews.
To expand the pool of reviewers among junior scholars, it seems to me that there needs to be a database set up of potential reviewers. This would have to have cvs and samples of the person’s own publications/writing. Does anyone have an idea about how to get such a thing going?
Edit: Just got a note from a managing editor of a major journal that is relevant to this discussion: “I wonder if > 3 reviews is happening for some journals because they’re using the “automatic invitation” feature that comes with most of this new tracking software we’re all using (you load up the list of potential reviewers with a bunch of names and the database automatically invites the next person on that list if any of the previous reviewers are late — we declined to do this and have set up our system to be far less automated).”
Scatterplotters: Your thoughts?