I am working on a revise-and-resubmit for a comment on a paper published in a prominent journal. Believe me, I am by this point so over being surprised by incompetent work appearing in prestigious places, and yet this paper was beyond the beyonds. The paper has three sections that claim to offer separate scientific contributions. Regarding one of them, here is a sentence I just drafted:
In sum, [author]’s analysis of [thing] treats arbitrary survey categories as natural distinctions, presents overall results that are logically necessary as though they were empirical findings, and characterizes specific results in ways that contradict simple cross-tabulation of the pertinent variables.
The kicker? This sentence is about the section of the original paper that aggravates me least.
Even so, the only reason I wrote a comment on this paper in the first place is that the paper was to me indicative of a Larger Problem, and I wanted to call attention to the Larger Problem. Unfortunately, the upshot of the editor’s directive regarding my original submission is to remove discussion of the Larger Problem–while encouraging me to write a whole paper just about that, which I may do–and instead in this comment focus narrowly on how this particular paper is incorrect and incompetent. Which isn’t nearly so rewarding intellectually, and isn’t exactly enjoyable work either.
I mean, the author of this paper plainly has a poor grasp on what s/he’s doing statistically. S/he also does something in the paper that I personally regard as disingenuous, or ignorant to the point of negligence (I’ll let you choose which of these is less harsh). But, other than the disingenuousness issue, it’s not the author’s fault that her/his incompetent work made it into a prominent journal. I wish in cases like this the names of the reviewers of the paper were listed, and that they were given the opportunity to defend their recommendation. I feel bad because I imagine this person sitting in their office, beaming with pride about their prominent publication, not knowing that I am sitting here in my own office working on articulating exactly why this person’s object-of-pride should be instead understood as an scientific atrocity that never should have made it through peer review. (And so close to Christmas, to boot.)
I thought about not doing the R&R if it couldn’t be about the Larger Problem. Gore Vidal said that one should never pass up the opportunity to have sex or be on television, and the academic equivalent of that aphorism would be the chance to have one’s work appear in a prominent forum (a comment included). But since I don’t think the saying is really true about sex or TV, I’m not sure why I would buy its academic extension. Instead, the best I can explain why I’m pressing ahead with revision is that the existence of this paper genuinely enrages me–as I was reading through part of it last night, my hands started shaking–and I can’t let it go unchallenged. Ugh.
scatterplot 
I bet that, in addition to these problems, the paper also had capital letters in its title. I know how that gets to you.
LikeLike
The author also didn’t use a jump even though the paper went on for longer than a screen. Grrrr!
LikeLike
Personally, I don’t see the point of the jump. It drives me crazy to have to add that extra click–and especially to have to click through from RSS if I’m not planning to comment.
LikeLike
This is a tension between RSS readers and people who read by just opening the blog in their browser. The issue for the latter is that people like to know who the author of a post is before they start reading a post, and also people don’t like having to scroll down far to read, say, the third post on our page. But it’s annoying for RSS readers. I’m trying to figure out if there is a way to syndicate RSS without the jump.
LikeLike
To repeat something you said to me last year, when I was feeling grumpy about having spent a lot of time on a review for a manuscript that I thought was abysmal (and on my and other’s recommendation was rejected): “You are doing your part to uphold standards in the discipline.”
LikeLike
Sara: Yes, but it’s so much better to have these things taken care of in the review stage.
I would have this comment basically done, but there is this part of the paper where I can’t figure out what the author did, and I think it might be because the description in the paper doesn’t actually make sense. It’s hard to figure out whether something objectively doesn’t make sense or whether it does make sense and you just don’t get it.
LikeLike
did the author use a widely available dataset you could replicate?
LikeLike
Of course, it’s better for everyone for these things to be taken care of early on! As you may recall, I was upset that the paper-in-question made it through review at Prominent Journal, in part, because it undermined my (perhaps naive?) faith in peer review.
As for whether the part of the paper doesn’t make sense or is just incomprehensible – why don’t you show it to a colleague and see if someone else “gets” it?
LikeLike
GR: Yes, the paper uses a dataset I have.
Sara: Yes, right. I’m working on this. The problem is that sociology is not exactly overabundant with people who understand the method that this person is using in the part I can’t figure out. (Although, seemingly the only people who persist in thinking this is an effective way of doing analysis are, in fact, in sociology. If we use the word ‘cult’ in the sense of ‘cult’ movies rather than Branch Davidians ‘cult,’ it’s kind of a cult method of quantitative data analysis.)
LikeLike
Five dollars to the first person who can “call” the cult method!
LikeLike
Oooh, Scatterplot needs our equivalent of a JFW kewpie doll! What shall we give as a reward for our contests (besides shakha’s money?)?
LikeLike
what is bayesian?
LikeLike
Bayesian methods are not a kind of cult analysis.
LikeLike
The only ‘cult’ type method that I’m familiar with is QCA, and since it has qualitative in the title, I’m thinking that’s not it.
This is probably the one time that I’ve been glad that I’ve never published a sole-authored piece in a prominent journal, and therefore am not “sitting in [my] office, beaming with pride about [my] prominent publication,” because, well… just because.
LikeLike
Of course one uses quantitative data…
LikeLike
Of course. Of freaking course. I just realized what I am going to have to do to figure out what this person did. I am going to have to… ugh, I can’t believe I am typing this… import the data into SPSS and work from there. No way I’m going to be able to figure out what this author by trying to reconstruct the analysis in Stata, but if I could see the menus he was pointing and clicking for this, maybe.
Here at Northwestern you can only use SPSS on Linux. No freaking way I am doing that. No freaking way.
LikeLike
is the “cult” method SEM?
could you write to the author and ask him/her what he/she did?
LikeLike
Spelunking into SPSS helped make it more clear, although this one part I’ve concluded in contradictory in the text. Literally, a figure presents a results, the text describes the figure in a way that is inconsistent with the y-axis of the figure, and then the text later describes the figure in a way that is consistent with the y-axis but inconsistent with the x-axis.
The ‘cult’ method is not SEM.
LikeLike
2SLS?
LikeLike
2SLS is an instrumental variables estimator. Instrumental variables estimators are so fetishized in economics the past few years that there have been magazine articles about it.
E-mail me and I’ll tell you the answer. I don’t know why I’m squeamish about saying it here.
LikeLike
So you are sure that the object of your comment is not reading this blog?
LikeLike
OW: Of course, I have considered this. My feeling is that I don’t care. (Fellow Scatterplotters may disagree.) If the comment is accepted, the person will be sent a copy, and anybody who reads the comment will be able to discern that I think that the analyses conducted in the paper are quite badly done and that the set-up for the paper contains a key argument that seems disingenuous. I’m not sure what I will do with the comment if it is rejected.
What I think would be weird and quasi-manipulative would be writing the post knowing that the paper’s author was a regular reader or something. I would not do that.
Negative comments about other’s work are delicate matters anyway, as I know there are many people in sociology who think nothing about the substance of the discipline is worth making an enemy over.
LikeLike
The only comment I’ve ever published was solicited by the editor because the article included a critique of my work. I wrote a critique of his critique and his analysis, and he wrote a response saying my critique was invalid. I still thought I was right, he thought he was. We sent drafts of our comments to each other and communicated back stage and remained professional friends.
I was involved in a similar round when I was one of the authors of an article that criticized others; they were offered the chance to write a rebuttal and then we responded. This got a little testier due to some exchanges that did not involve me, but again, backstage communication softened most of the animus.
I have no experience publishing critical comments about people I don’t know.
LikeLike
I think it’s easier to be professionally friendly when the dispute provoking a comment is on a relatively high-level point of debate. One can see from my CV that I’ve had comment-reply before tangles with a couple people, and (to my knowledge) I have had perfectly amicable relations with those people. When it’s more like, the premise of this paper seems misleading and the research is badly done, I’m not sure what the prospects are there. Maybe this is just me channelling my inner Wacquant.
In any case, with respect to the issue that some sociologists completely turn off their critical thinking skills in the face of any work that purports to document the triumph of the so-called “social” over the so-called “biological,” by this point I’ve reached a weariness where I am probably spoiling for a fight anyway.
LikeLike