I’m looking for examples of published studies that are severely underpowered. What it means for a study to be severely underpowered is that it requires an implausibly large effect size in order to be publishable as a positive finding.* I’m kicking around a paper idea about certain aspects of the problems with underpowered studies, which will be easier to explain if I have three actually published examples. I’m currently at least one short.
The hurricane name study is one example. With the study, I think a power analysis could have been done indicating that the only way the study could produce a publishable positive result is by finding that most of the lives lost in severe hurricanes named after women would have been saved if the hurricanes were named after men. If so, then if a scientist regards that as implausible, it doesn’t make much sense to proceed with a study. The most pernicious thing about underpowered big effects is that even when people recognize that the estimated effect size is implausible, they still think the results mean a plausibly-sized real effect is more likely than no effect at all. This is not completely wrong, but it’s so close that it’s best just to think: wrong.
If you aren’t sure how tell if a study is badly underpowered, the simplest diagnostic is that the estimated coefficient seems very big and yet the p-value of the study is not very small. If you are squeamish about suggesting a study in the comments, feel free to e-mail: jfreese at northwestern
* In principle, severely underpowered study with null results might be published as a debunking of something. In this case, that the study is underpowered means that an implausibly large effect size would have been needed for the study not to succeed in producing a null finding, so in that respect it’s hardly much of a debunking. If any examples of that come to mind, definitely let me know.