sociology news

From a section on concerns raised in the Annual Report of the American Sociological Association’s methodology section (Ross Stolzenberg of the University of Chicago, author):

First, the American Sociological Review is widely believed by section members to have an explicit policy against publication of methodological articles, and to be further biased against papers with advanced methodological content. Section members point out that the ASR has no policy against publication of papers from any other topic area in the discipline, that papers in other topic areas of sociology are published in both the ASR and topic-oriented ASA journals, and that influential methodological articles are among the most widely-cited papers in the discipline. Exclusion of widely-cited papers from publication in a journal is self-destructive for the journal, of course, because those papers contribute positively to the journal’s so-called impact score, which is now so popularly used to evaluate journal quality.

Second, section members voiced distress at an apparent decline in the quality of journal evaluations of the methodological content of papers submitted to ASA journals. Members recited lurid tales of rejection letters or editorial demands for revision based on the evaluations that denied facts of simple algebra and elementary statistics, as well as more advanced topics. Some members expressed embarrassment for their discipline, and described the frustration of colleagues whose papers had been rejected for bad cause.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

5 thoughts on “sociology news”

  1. I’m still fairly new at this, but am beginning my stack of rejection letters. I have to say, SPQ did send me probably the single best rejection letter ever written. And I got great rejection comments on a paper to AJS too. Although I did get a rejection letter that simply said that my sample size was too small (even though I had 100% of the population), so eh?

    Like

  2. I’m actually on the ASR editorial board, which just means I am asked to review lots of articles (about 8 per year) – I don’t make decisions or write rejection letters! But from my experience, and not representing anyone else, a couple observations:

    1. As ASR has published more research using more sophisticated statistical methods – partly as a result of the rise in average computing power – more methodologists complain that its methodological standards are dropping. Why?

    2. I have never heard of an explicit (or implicit) policy against methods articles in ASR. The last couple editors, at least, have said that ASR aims to publish articles that are comprehensible and interesting to all (or most) sociologists. Methods articles have a hard time hitting that note, or so reviewers and editors believe. Why?

    3. Sociologists have become more specialized in the last few academic generations – hence all the different journals. As methods have become more sophisticated, they have become harder for other sociologists to understand. I reckon that a smaller proportion of cutting-edge methods articles today can be read and understood widely than could have been in the past. The same is probably true of other specialties in the discipline, too.

    4. If by “methodological articles” the section critics are mostly referring to “statistical methodology articles,” then I have a question: Do the flagship journals of other disciplines’ national associations publish articles on statistical methodology? Maybe that’s not what they mean.

    -Philip Cohen

    Like

  3. It’s interesting to see this complaint coming from the quant methods side of the spectrum. Philip points to some important questions.
    On point 1, I think this ia a problem of increasing potential diversity of reviewers. The success of high-end statistical methods means that there are many users, and many of us may not fully appreciate the contributions of true methodologists. They may have a legitimate gripe.
    On 2, I think editors at top general journals should be bird dogging for top methodological articles. True contributions get cited. And, every time I check the Soc. Methodology box for my second journal, ASA never sends it–I pay extra and get nothing.
    For 3, I could make the opposite argument. Now, we have sophisticated methodologists working in crime, organizations, geography, labor, politics, family, etc. Each area has unique problems, and often the unique solution for one field has considerable crossover into other areas. The whole area of contextual glm models is a great example.
    For 4, It’s be interesting to actually chart that out as a question.. There are more jouranls in Psych and Econ, but both also publish advances in top journals–and both also have several prominent specialty journals for methods.

    Like

  4. Although I am well acquainted with many in the methodology section, I was not a participant in the discussion Rafe references.

    In my view, point 1 of Rafe’s message is a little bit silly. I very much doubt that there are many methodology pieces that are submitted to ASR, as there are other venues far more suitable. Both SM and SMR are great outlets already, and they allow methodology pieces to be written as the authors want them to be written — with all of the details that ASR could never spare the pages to print. They also have Editors who are experts in methodology, and this results in a review process that actually improves papers as they go through the pipeline.

    I think the methodologists registering this paticular complaints are really more upset at the journal article hierarchy as they see it. They feel that ASR articles are granted too much prestige, and they are on average of lower quality than articles published in methods journals. From my perspective, they are apples and oranges, and the quality of work can be judged without regard to publication outlet. What is important is that job search processes and promotion reviews not be unreasonably influenced by article placement. One should never allow a colleague to get away with statements like: “I know the article is not very good, and probably wrong. But it will be influential because it is in AJS, and so we should hire job candidate ____.”

    Rafe’s second point, however, is not silly and contains a very important component. Too many articles published in ASR (and many other worthy sociology journals) are based on fancy statistical models that the authors have figured out how to get the computer to spit out. They are not, however, properly interpreted, either with respect to statistical inference or causal inference. Coefficients with asterisks are too often given causal interpretations with no recognition of the current literature on causality that shows clearly why they do not deserve such interpretations. The substantive sizes of coefficients are too rarely considered, and coefficients that fall just below p-value cutoffs are mistakenly interpreted as if they are zero. This is all horribly embarrassing for sociology, and we have now fallen well behind economics and political science in our capacity as a discipline to properly analyze data.

    Who is to blame for all of this? Lots of people.

    Like

  5. My very loose connection to the current SM editor suggests to me that SM is trying to be a broad methodological journal and not just quantitative. And, I think, the past volumes attest to this.

    That being said I believe there are moderate positive correlations between the complexity (and newness) of the statistical model, its probability of getting published (in any non-methods journal) and said model being used or interpreted incorrectly. I would love to see the paper that tried to test those hypotheses.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.