the glamorous life

I am working on a revise-and-resubmit for a comment on a paper published in a prominent journal. Believe me, I am by this point so over being surprised by incompetent work appearing in prestigious places, and yet this paper was beyond the beyonds. The paper has three sections that claim to offer separate scientific contributions. Regarding one of them, here is a sentence I just drafted:

In sum, [author]’s analysis of [thing] treats arbitrary survey categories as natural distinctions, presents overall results that are logically necessary as though they were empirical findings, and characterizes specific results in ways that contradict simple cross-tabulation of the pertinent variables.

The kicker? This sentence is about the section of the original paper that aggravates me least.

Even so, the only reason I wrote a comment on this paper in the first place is that the paper was to me indicative of a Larger Problem, and I wanted to call attention to the Larger Problem. Unfortunately, the upshot of the editor’s directive regarding my original submission is to remove discussion of the Larger Problem–while encouraging me to write a whole paper just about that, which I may do–and instead in this comment focus narrowly on how this particular paper is incorrect and incompetent. Which isn’t nearly so rewarding intellectually, and isn’t exactly enjoyable work either.

I mean, the author of this paper plainly has a poor grasp on what s/he’s doing statistically. S/he also does something in the paper that I personally regard as disingenuous, or ignorant to the point of negligence (I’ll let you choose which of these is less harsh). But, other than the disingenuousness issue, it’s not the author’s fault that her/his incompetent work made it into a prominent journal. I wish in cases like this the names of the reviewers of the paper were listed, and that they were given the opportunity to defend their recommendation. I feel bad because I imagine this person sitting in their office, beaming with pride about their prominent publication, not knowing that I am sitting here in my own office working on articulating exactly why this person’s object-of-pride should be instead understood as an scientific atrocity that never should have made it through peer review. (And so close to Christmas, to boot.)

I thought about not doing the R&R if it couldn’t be about the Larger Problem. Gore Vidal said that one should never pass up the opportunity to have sex or be on television, and the academic equivalent of that aphorism would be the chance to have one’s work appear in a prominent forum (a comment included). But since I don’t think the saying is really true about sex or TV, I’m not sure why I would buy its academic extension. Instead, the best I can explain why I’m pressing ahead with revision is that the existence of this paper genuinely enrages me–as I was reading through part of it last night, my hands started shaking–and I can’t let it go unchallenged. Ugh.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

27 thoughts on “the glamorous life”

  1. Personally, I don’t see the point of the jump. It drives me crazy to have to add that extra click–and especially to have to click through from RSS if I’m not planning to comment.

    Like

  2. This is a tension between RSS readers and people who read by just opening the blog in their browser. The issue for the latter is that people like to know who the author of a post is before they start reading a post, and also people don’t like having to scroll down far to read, say, the third post on our page. But it’s annoying for RSS readers. I’m trying to figure out if there is a way to syndicate RSS without the jump.

    Like

  3. To repeat something you said to me last year, when I was feeling grumpy about having spent a lot of time on a review for a manuscript that I thought was abysmal (and on my and other’s recommendation was rejected): “You are doing your part to uphold standards in the discipline.”

    Like

  4. Sara: Yes, but it’s so much better to have these things taken care of in the review stage.

    I would have this comment basically done, but there is this part of the paper where I can’t figure out what the author did, and I think it might be because the description in the paper doesn’t actually make sense. It’s hard to figure out whether something objectively doesn’t make sense or whether it does make sense and you just don’t get it.

    Like

  5. Of course, it’s better for everyone for these things to be taken care of early on! As you may recall, I was upset that the paper-in-question made it through review at Prominent Journal, in part, because it undermined my (perhaps naive?) faith in peer review.

    As for whether the part of the paper doesn’t make sense or is just incomprehensible – why don’t you show it to a colleague and see if someone else “gets” it?

    Like

  6. GR: Yes, the paper uses a dataset I have.

    Sara: Yes, right. I’m working on this. The problem is that sociology is not exactly overabundant with people who understand the method that this person is using in the part I can’t figure out. (Although, seemingly the only people who persist in thinking this is an effective way of doing analysis are, in fact, in sociology. If we use the word ‘cult’ in the sense of ‘cult’ movies rather than Branch Davidians ‘cult,’ it’s kind of a cult method of quantitative data analysis.)

    Like

  7. The only ‘cult’ type method that I’m familiar with is QCA, and since it has qualitative in the title, I’m thinking that’s not it.

    This is probably the one time that I’ve been glad that I’ve never published a sole-authored piece in a prominent journal, and therefore am not “sitting in [my] office, beaming with pride about [my] prominent publication,” because, well… just because.

    Like

  8. Of course. Of freaking course. I just realized what I am going to have to do to figure out what this person did. I am going to have to… ugh, I can’t believe I am typing this… import the data into SPSS and work from there. No way I’m going to be able to figure out what this author by trying to reconstruct the analysis in Stata, but if I could see the menus he was pointing and clicking for this, maybe.

    Here at Northwestern you can only use SPSS on Linux. No freaking way I am doing that. No freaking way.

    Like

  9. Spelunking into SPSS helped make it more clear, although this one part I’ve concluded in contradictory in the text. Literally, a figure presents a results, the text describes the figure in a way that is inconsistent with the y-axis of the figure, and then the text later describes the figure in a way that is consistent with the y-axis but inconsistent with the x-axis.

    The ‘cult’ method is not SEM.

    Like

  10. 2SLS is an instrumental variables estimator. Instrumental variables estimators are so fetishized in economics the past few years that there have been magazine articles about it.

    E-mail me and I’ll tell you the answer. I don’t know why I’m squeamish about saying it here.

    Like

  11. OW: Of course, I have considered this. My feeling is that I don’t care. (Fellow Scatterplotters may disagree.) If the comment is accepted, the person will be sent a copy, and anybody who reads the comment will be able to discern that I think that the analyses conducted in the paper are quite badly done and that the set-up for the paper contains a key argument that seems disingenuous. I’m not sure what I will do with the comment if it is rejected.

    What I think would be weird and quasi-manipulative would be writing the post knowing that the paper’s author was a regular reader or something. I would not do that.

    Negative comments about other’s work are delicate matters anyway, as I know there are many people in sociology who think nothing about the substance of the discipline is worth making an enemy over.

    Like

  12. The only comment I’ve ever published was solicited by the editor because the article included a critique of my work. I wrote a critique of his critique and his analysis, and he wrote a response saying my critique was invalid. I still thought I was right, he thought he was. We sent drafts of our comments to each other and communicated back stage and remained professional friends.

    I was involved in a similar round when I was one of the authors of an article that criticized others; they were offered the chance to write a rebuttal and then we responded. This got a little testier due to some exchanges that did not involve me, but again, backstage communication softened most of the animus.

    I have no experience publishing critical comments about people I don’t know.

    Like

  13. I think it’s easier to be professionally friendly when the dispute provoking a comment is on a relatively high-level point of debate. One can see from my CV that I’ve had comment-reply before tangles with a couple people, and (to my knowledge) I have had perfectly amicable relations with those people. When it’s more like, the premise of this paper seems misleading and the research is badly done, I’m not sure what the prospects are there. Maybe this is just me channelling my inner Wacquant.

    In any case, with respect to the issue that some sociologists completely turn off their critical thinking skills in the face of any work that purports to document the triumph of the so-called “social” over the so-called “biological,” by this point I’ve reached a weariness where I am probably spoiling for a fight anyway.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.