Just in time for Hallowe’en, Phil Cohen has posted an account of a recent experience of trying to publish an article. The account is more striking when one pauses to think that the story is not getting told because it is extreme in a discipline-wide sense, but that it’s extreme for one of the few folks who write blog posts about things like this. In other words, too many people with too many papers are ending up with these sort of stories.
I appreciated Phil’s forthrightness in the account, particular the part where he reproduced one editor’s request to insert citations to more papers from their journal.
Beyond that, I was particular fond of this paragraph of the summary:
Sociologists care way too much about framing. Most (or all) of the reviewers were sociologists, and most of what they suggested, complained about, or objected was about the way the paper was “framed,” that is, how we establish the importance of the question and interpret the results. Of course framing is important – it’s why you’re asking your question, and why readers should care (see Mark Granovetter’s note on the rejected version of “the Strength of Weak Ties”). But it takes on elevated importance when we’re scrapping over limited slots in academic journals, so that to get published you have to successfully “frame” your paper as more important than some other poor slob’s.
scatterplot 
Yes! I was thinking to myself, and have increasingly been thinking, that our discipline has too little interest in generating a body of empirically well-verified facts and too much interest in spin [aka framing] and novelty. And at times too little ability to evaluate the accuracy of results, which is probably due to our methodological diversity.
LikeLiked by 2 people
One possibility is that the reason for so much emphasis on framing by reviewers is that details of results are less scrutinized.
LikeLiked by 2 people
Oddly enough, my peer review stories are even more horrifying, see http://www.ethnography.com/fatuous-naive-and-bold-at-the-same-time-welcome-to-the-wonderful-world-of-peer-review/ A publication timeline of 2012-2015 seems like greased lightening in the context of some of my experiences.
Nevertheless, you make good points about the narrowness of peer review. Too often it squeezes out the author’s originality, jokes, etc., in the interest of satisfying today’s disciplinary fads. This is unfortunate, because it makes research into a backwards looking over-the-shoulder matter rather than a forward looking one.
Having said that, I still find the process appropriate for certain kinds of writing. My feelings are that there are other venues for things that are not so nastily peer reviewed–book writing comes to mind. Sure you never get into the “big name” journals, but in fifteen or so years on promotion committees I have never heard of that being a problem. There is simply little interest in “where have you published” at comprehensive MA granting institution where I teach. Rather, it is have you published, and is it interesting?
As for why I still like peer review–I do get the most incisive review of my papers via peer review. The feedback can be mean and nasty, but it can also be quite incisive. Which I guess is why I swing back and forth between different venues–books, peer reviewed articles, encyclopedia entries, and so forth. All should be considered part of the complete academic package, particularly in a field like sociology.
LikeLiked by 1 person
Framing is an essential aspect of a paper. Science is not just a set of facts, but a process of connecting facts together to build understanding. Moreover, some framings are more convincing than others partly because they are more strongly rooted in a set of established facts. Framing is fair game in peer review.
I’m writing this not because I disagree with Philip Cohen’s blog. I’ve had a paper rejected because it addressed a topic that was no longer in vogue even though it is still an important and unresolved question. The situation was made worse by the fact that the imperfect analyses produced results that did not fit well with currently preferred theories/explanations. So, I agree that it is unfortunate when reviewers are looking for arguments that support dominant narratives in the field. The review of Lykke and Cohen’s statistical analysis for the paper was beyond aggravating, but funny in a glad-it-isn’t-currently-happening-to-me kind of way. It rings true; more so in my experience as author than editor or reviewer, of course.
I’m writing this because I fear that some will interpret this critique to mean that reviewers should not consider framing. It is unfortunate when reviewers attend to the framing without sufficient attention to the quality of evidence. Sometimes this happens because the reviewer is so unconvinced by the conceptualization that they are recommending rejection even before getting to the empirical parts. That is fine as is critique focused on a mismatch between conceptualization and analysis. But it is a concern when a reviewer decides she likes the framing/argument, does not consider the quality of the data, and recommends acceptance. But here I’m calling for more rejections, not an easier review process.
Framing is fair game in peer review, but we should have a conversation about standards for framing. It’s bad when science becomes a popularity contest, whatever status game you’re in.
Finally, if you review for JMF, please do not ask that authors include more citations to articles in JMF. If there is a specific relevant cite that happens to be published in JMF, then by all means…
LikeLiked by 3 people
I agree framing is important. But we are too hung up on it. Partly that is because we (the discipline) want every empirical analysis to make a “theoretical contribution” in order to be publishable. Which is great, when it’s real. But a lot of the theory-inspired claims in sociology journals are attempts to puff up an empirical finding that doesn’t need it.
We need better ways to get empirical findings on the record without elaborate stories before and after them in the paper.
LikeLiked by 3 people
I’ve always been a fan of the brief empirical report with a short introduction that explains the contribution of the analysis to understanding.
LikeLiked by 4 people