the shock and awe memo

Comment by a well-known sociologist and friend on Facebook:

“We have turned from a discipline of article writers into a discipline of revision memo writers. This is a very sad thing.”

This concerns me, too. But what can be done? I’ve wondered if memos could be capped, but not sure if that would end up as routinely ignored as the page limits on ASA papers.

Thing is, my belief–and I’m very confident I’m not alone in this–is that long memos work. I’ve even said things like, “You need to write a shock-and-awe memo” (along with doing the actual revisions to the actual paper). But of course I think the discipline as a whole would benefit by maximizing the research time spent on the research products themselves, and not unpublished memoranda about those products.

UPDATE: An obvious answer to “What can be done?” is “Reduce the number of R&Rs” and/or “Promote non-R&R-giving outlets like Sociological Science and As Yet Untitled.” And, yes. But, is that it?

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

14 thoughts on “the shock and awe memo”

  1. The problem is it’s a signaling game with an arms race dynamic. The way you signal: longer response memos and supplementary appendices, but the bar for what impresses people for how thorough you are keeps getting raised. Also, reviewers’ demands keep growing. AJPS in poli sci has an 8500 word cap on response memos and the problem is that it’s very hard to explain how you satisfied all of the comments from four long reviews in that space.

    Like

  2. Neil Gross doesn’t even send out the revision memo to reviewers — a practice many find frustrating, but his view is that if the revision doesn’t stand on its own, the revisions aren’t adequate. And I think it follows from that view that a decision not to pursue a particular path should be noted in a footnote or similar.

    Liked by 1 person

  3. I’m on the record as saying that we need to think not only about what editors and authors can do about this but that we need to change the culture of reviewing so that reviewers learn to tell the difference between “needs to” versus “could have.”

    Of course the trick is that cultural explanations are often more of gripes than solutions since structural explanations imply structural solutions. As Moynihan said “The central conservative truth is that it is culture, not politics, that determines the success of a society. The central liberal truth is that politics can change a culture and save it from itself.” The question is what would a “liberal” editor do about this problem in order to save the culture of peer review from itself?

    I have to admit that I don’t really know, but a few possibilities suggest themselves:

    — create purely evaluative journals like Sociological Science, both for the direct option value and to scare straight the incumbents. The Menshevik version of this is to limit to just one round of R&R.
    — the Gross model of not sharing the memos with reviewers
    — explicit instructions to reviewers to focus on “needs to” versus “could have” and cover letters that give the authors permission to ignore certain issues
    — page limits on response memos that have the effect of limiting authors to discussing general trends rather than item by item

    Liked by 1 person

  4. I think that we can do a better job teaching graduate students how to write an effective memo. I am not sure the over the top “shock and awe” memo is actually the most effective. I think reviewers are mainly looking for guidance and direction on how the author addressed their particular concerns (which seems reasonable to me). I think that in most cases it is sufficient to just say something like “reviewer A raised a question about X. I addressed this issue on page 17 of the revised manuscript.” I think the lengthier discussions in the reviewer memo should be reserved for those points where you actually disagree with the reviewer and need to provide a justification for sticking to your guns.

    Like

  5. Not surprisingly I see the main solution as being structural, precisely because the incentives are skewed, as Jeremy points out: collectively irrational to write a longer memo than paper, but individually rational since “long memos work.” (Structural solutions don’t have to mean starting new journals, I tell myself late at night, but could involve making memos for the editor’s eyes only.)

    That said, Gabriel is right that this is a manifestation of a larger cultural issue that treats the review process as an outlet for being heard and (most importantly) for getting respect/being shown deference. Brayden has a great example in the thread on FB of a reviewer saying “I also think it is common courtesy for authors to take the time to address all of the reviewer comments in a response letter.” This is slightly different from Gabriel’s point, in that it is not just about “you could do this, too” but rather “did you pay attention to me, or was my hard work for naught?” (A colleague recently had a new reviewer complain because the short revision memo did not exhibit sufficient deference to the *other* reviewers, which seems an extreme manifestation of viewing the review process as a deference game.)

    The problem with insisting on deference is that reviewers have the power to insist (especially when editors allow them), and have strong incentives to do so, since to do otherwise is to entertain the idea that their comments were off-base, irrelevant, or just wrong. It does seem like it would be better if we could admit that we get things wrong.

    Liked by 3 people

    1. Jesper’s comment provides an excellent reason for the Neil Gross model of not sharing the memo with the reviewers. (The best argument against it is Andy’s suggestion that it will lead to a bunch of non sequitur footnotes).

      I recently published a piece in Sociological Theory and I was happy with the policy but I didn’t find out about it until after the R&R. If I had known ex ante, I probably would have spent half as much time (and tact) on the memo precisely for the reasons Jesper is saying. I think Neil would understand if I responded to a particular comment with a single plain English sentence saying “reviewer X wants me to cite Smith 1975, but I can’t cite that as it’s crap and besides my article is about the Hollywood studio system and Smith 1975 is about leatherback turtle hatchlings,” whereas if I think reviewer X himself/herself will read it I can’t just say “yeah, not gonna do that,” without first going through 400 words of ritual proskynesis, kow-towing, and good old fashioned brown-nosing lest reviewer X (or even reviewer Y on reviewer X’s behalf) be offended at such an outrageous lack of supplication from such a social inferior as an author.

      Basically my reaction to learning about the ST policy after I had written my (44 page) response memo was:

      Liked by 1 person

  6. Structural pressures of the sort described above could also lead to straightforward disciplinary fragmentation. The pressure of trying to please too many reviewers with different quality standards pushes things so that Gondwanaland breaks up into smaller units whose inhabitants agree on what and how to assess things. Those who edit and publish in elite journals have some interest in containing the tectonic forces, so to speak, but typically this is achieved by being able to enforce a hard line on some combination of acceptable methods, theories, and topics. If a field lacks that sort of consensus—or the resources to enforce one—it seems to me that it will eventually pull itself apart into subunits that can get along with each other. (The examples of success and failure in this regard in cognate fields seem pretty clear.)

    Editorial interest in containing these pressures is has its limits, though. This is because, in the short run, if a field has relatively low methodological/theoretical consensus this will tend to increase the power of generalist journals and their editors, because publication in those outlets becomes one of the few generally agreed-on metrics of quality amongst people who otherwise disagree about a lot of things. But in the long run the centripetal forces seem likely to work against the discipline as a coherent entity, and thus the generalist journals that claim to reflect it, as well.

    Right now, the Shock-and-Awe memo is one kind of safety valve, but we’ve already seen high-status actors willing to establish alternative models. If those take hold as publication outlets, one wonders whether other anti- or alt-disciplinary organizational innovations can be far behind, whether originated by (currently) higher- or lower-status groups.

    Like

  7. For those of you who don’t have the right network connections, I’m the “well-known sociologist” who is FB friends with Jeremy. And I stand by what I wrote. I think revision memos suck, even though I admit I’ve been impressed by a few. I have, however, been MORE impressed by articles that came back from revision with changes that had nothing to do with what the reviewers suggested but represented the evolution of the writer’s own thinking and work, and have been very unimpressed with revisions in which the writers did exactly what the reviewers asked and nothing else. I also, early in my career, published several highly-regarded articles in which I had made changes between the initial submission and the revision that made the article much better that had nothing at all to do with what the reviewers asked for, and had to negotiate the problem of reviewers who really did not know what they were talking about.

    I ESPECIALLY loathe the instruction to send back a revised article with track changes to show how you responded to the reviewers.

    Liked by 1 person

  8. How common is the shock-and-awe memo, really? I received one recently — it was literally longer than the paper — but most of the memos I get are 5 pp. or less, with almost none above 10 pp. This is across a pretty broad range of journals, I think. Are they specific to a certain type of article?

    That’s not to say that such memos aren’t a waste of time. I’m just curious if I’m an outlier. Or maybe they’re more common in other subfields.

    Like

  9. While we are complaining, I submitted an article to an interdisciplinary health journal that was ~4,000 words long. The response memo that I ended up writing was 18 pages based on seven pages of review comments (many of which asked for analyses to be redone, which required their own explanation of the methods and measures). This did not bother me (that much) — the suggestions were helpful, the reviewers were obviously engaged and interested in the project, and there was a productive dialogue.

    What did irritate me, however, was the 1.5 additional pages of comments the reviewer asked for AFTER the 18-page response memo. On a now ~5,500 word article.

    Liked by 2 people

  10. The request to track changes in the document is quite common in psych and interdisciplinary public health journals. The upside is that there also seems to be a norm of reviewers asking for relatively minor changes rather than a request to rethink your whole argument or interpretation of findings. In my experience, these journals also want the revisions in 4-6 weeks and reviews come in 6-8 weeks. So you get the speed many of us want. I don’t know that you get better papers and, generalizing again from my limited experience, you’re less likely to really learn something or push your thinking forward in a new direction with a review process like that.

    Liked by 1 person

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s