the dilemma of the subordinate methodologist

I’m not sure if the author of this post is a graduate student or undergraduate, but I found it an intriguing statement about the problem younger people interested in methodology can find themselves in while working with established people who are very steeped in conventional practices and productivity. Quote:

One thing that never really comes up when people talk about “Questionable Research Practices,” is what to do when you’re a junior in the field and someone your senior suggests that you partake. […] I don’t want to engage in what I know is a bad research practice, but I don’t have a choice. I can’t afford to burn bridges when those same bridges are the only things that get me over the water and into a job. (HT: @lakens)

Mostly this is just a statement about power.* But it’s also maybe a statement about what can happen when developments allow the possibility of radical doubt to settle upon a field. Normally a junior person can have methodological doubts, but still think, “Well, these people must know what they are doing, because it’s been successful for them and so ultimately in practice it works, right?” But what happens when you have developments that lead to a lot of people starting to whisper and murmur and talk about how maybe it doesn’t work?

* I mean power in the ordinary sociological sense, not my ongoing obsession with statistical power.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

9 thoughts on “the dilemma of the subordinate methodologist”

  1. This person faces the same dilemma Deirdre does and critical methodologists generally: jumping to the ethical issue. Scolding and indignation rarely move an equilibrium of belief (I should know).

    I think the real problem is sociological here, not ethical. Significance testing reduces the dimensions of the decision space, at least for most people, to a binary outcome. That unthinking heuristic, like all institutional rules, is computationally efficient for the individual and institutionally stable because it’s an easily understood purity boundary.

    And historically, the situation in which more sophisticated statistics are being widely used by people who don’t understand the math behind them is only as recent as computers (not slamming; I’m one of those people). So this problem will I think slowly abate as statistical discussion increases in competition, and new institutional heuristics begin to replace asterisk hunting.

    Interestingly, at that point, people will be applying that new heuristic in no more of a thoughtful manner – we’ll just be applying a smarter institution.

    Like

  2. Weren’t junior people also caught up in the Stapel scandal? Stapel didn’t use out of date methods, he just made up data. The discussion about that case blamed the premium on novelty, as well as the power issues that kept junior scholars from probing where the data came from, because Stapel wouldn’t let them run their own experiments, just fed them data.

    Like

    1. Stapel was busted by graduate students — that’s a story in and of itself.

      But yes, there were other students who had their dissertations based on faked Stapel data. I don’t really get how that worked: these students devised experiments, and then the idea was that Stapel claimed he had access to some high schools or other experimental pools and didn’t wanted anyone else mucking with it, so he would just “collect the data” and analyze it himself.

      Like

  3. A few years back the Office of Research Integrity produced an interactive video to model appropriate responses to a situation where someone suspects someone of conducting fraudulent research (http://ori.hhs.gov/thelab). I thought it was interesting and helpful, if nothing else then to see how ORI’s are supposed to work.

    Like

  4. Just to be clear: my point was that status is a problem even in outright fraud. Now imagine there is no fraud involved, just a suspicion that the statistical methods are not well chosen. Why wouldn’t a lower-status person tiptoe carefully around the problem? And then, if you have been around long enough, you’ll realize the faddishness in the “correct” methods and how that mucks up judgment.

    Like

    1. Yeah, part of the issue here is what comprises “out of date methods.” In sociology, we might imagine that out-of-date methods would be like using listwise deletion when some young methodologist has been filled up the zeal of multiple imputations. For this psychology stuff, part of what’s at issue are practices that are very hard to defend, especially to the extent that people say they are often done and then not even reported in the article. So it could be that an outcome of this is improvement in reporting practices where something that people used to think was okay is no longer okay, which would make the previous way “out of date,” but it’s less clear whether it’s more properly thought of as methodologically outdated or morally outdated.

      Like

  5. Hey Jeremy, thanks for posting. To clarify, I am an RA in between undergrad and grad school. Here is the situation: we tested an intervention on groups X, Y, and Z. There are gains shown in X and Y, but not Z. Also, those gains in X and Y are only significant when we throw out an outlier from each group (and they want to collect a few more subjects to fill in for them). The proposal for the write-up is to only include X and Y, and to not mention Z or removed outliers.

    In regards to sophisticated stats and whatnot, we’re doing basic stuff that a freshman methods class covers.

    Like

    1. This is helpful context. I’m glad you posted the specifics of this. You are of course correct, not mentioning that the results are only significant if you drop outliers, and not mentioning that there was a whole other group run in the experiment that wasn’t mentioned, is misleading practice that social scientists should fight against.

      Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s