I’m not sure if the author of this post is a graduate student or undergraduate, but I found it an intriguing statement about the problem younger people interested in methodology can find themselves in while working with established people who are very steeped in conventional practices and productivity. Quote:
One thing that never really comes up when people talk about “Questionable Research Practices,” is what to do when you’re a junior in the field and someone your senior suggests that you partake. […] I don’t want to engage in what I know is a bad research practice, but I don’t have a choice. I can’t afford to burn bridges when those same bridges are the only things that get me over the water and into a job. (HT: @lakens)
Mostly this is just a statement about power.* But it’s also maybe a statement about what can happen when developments allow the possibility of radical doubt to settle upon a field. Normally a junior person can have methodological doubts, but still think, “Well, these people must know what they are doing, because it’s been successful for them and so ultimately in practice it works, right?” But what happens when you have developments that lead to a lot of people starting to whisper and murmur and talk about how maybe it doesn’t work?
* I mean power in the ordinary sociological sense, not my ongoing obsession with statistical power.
scatterplot 
This person faces the same dilemma Deirdre does and critical methodologists generally: jumping to the ethical issue. Scolding and indignation rarely move an equilibrium of belief (I should know).
I think the real problem is sociological here, not ethical. Significance testing reduces the dimensions of the decision space, at least for most people, to a binary outcome. That unthinking heuristic, like all institutional rules, is computationally efficient for the individual and institutionally stable because it’s an easily understood purity boundary.
And historically, the situation in which more sophisticated statistics are being widely used by people who don’t understand the math behind them is only as recent as computers (not slamming; I’m one of those people). So this problem will I think slowly abate as statistical discussion increases in competition, and new institutional heuristics begin to replace asterisk hunting.
Interestingly, at that point, people will be applying that new heuristic in no more of a thoughtful manner – we’ll just be applying a smarter institution.
LikeLike
Weren’t junior people also caught up in the Stapel scandal? Stapel didn’t use out of date methods, he just made up data. The discussion about that case blamed the premium on novelty, as well as the power issues that kept junior scholars from probing where the data came from, because Stapel wouldn’t let them run their own experiments, just fed them data.
LikeLike
Stapel was busted by graduate students — that’s a story in and of itself.
But yes, there were other students who had their dissertations based on faked Stapel data. I don’t really get how that worked: these students devised experiments, and then the idea was that Stapel claimed he had access to some high schools or other experimental pools and didn’t wanted anyone else mucking with it, so he would just “collect the data” and analyze it himself.
LikeLike
A few years back the Office of Research Integrity produced an interactive video to model appropriate responses to a situation where someone suspects someone of conducting fraudulent research (http://ori.hhs.gov/thelab). I thought it was interesting and helpful, if nothing else then to see how ORI’s are supposed to work.
LikeLike
Just to be clear: my point was that status is a problem even in outright fraud. Now imagine there is no fraud involved, just a suspicion that the statistical methods are not well chosen. Why wouldn’t a lower-status person tiptoe carefully around the problem? And then, if you have been around long enough, you’ll realize the faddishness in the “correct” methods and how that mucks up judgment.
LikeLike
Yeah, part of the issue here is what comprises “out of date methods.” In sociology, we might imagine that out-of-date methods would be like using listwise deletion when some young methodologist has been filled up the zeal of multiple imputations. For this psychology stuff, part of what’s at issue are practices that are very hard to defend, especially to the extent that people say they are often done and then not even reported in the article. So it could be that an outcome of this is improvement in reporting practices where something that people used to think was okay is no longer okay, which would make the previous way “out of date,” but it’s less clear whether it’s more properly thought of as methodologically outdated or morally outdated.
LikeLike
Hey Jeremy, thanks for posting. To clarify, I am an RA in between undergrad and grad school. Here is the situation: we tested an intervention on groups X, Y, and Z. There are gains shown in X and Y, but not Z. Also, those gains in X and Y are only significant when we throw out an outlier from each group (and they want to collect a few more subjects to fill in for them). The proposal for the write-up is to only include X and Y, and to not mention Z or removed outliers.
In regards to sophisticated stats and whatnot, we’re doing basic stuff that a freshman methods class covers.
LikeLike
This is helpful context. I’m glad you posted the specifics of this. You are of course correct, not mentioning that the results are only significant if you drop outliers, and not mentioning that there was a whole other group run in the experiment that wasn’t mentioned, is misleading practice that social scientists should fight against.
LikeLike
Thank you for your integrity. I hope you figure out a way to keep your it in the context you’re in.
LikeLike