Hurricane Name Study, how I wish I could quit you. But an inquiry prompted me to look some more. Again, you can download the data yourself and replicate their key model, Model 4 above, using the half-tweet’s worth of code I included earlier.
The table isn’t in the paper, only their supplemental materials. Notice there are two significant interaction effects: one for the dollar damage of a hurricane (highlighted in green), and the other for the minimum pressure (highlighted in yellow). Both are severity measures. You might think that because both coefficients have the same sign, they both are consistent with the story that, as hurricanes become more severe, the death rate goes up faster for “female hurricanes” than “male hurricanes.”
Hey, wait! Don’t more severe hurricanes have lower minimum pressure? Why, yes. You can confirm this several ways, but notice how the pink coefficient for the main effect of pressure has a negative sign. In other words, the green coefficient and yellow coefficient are actually telling you opposite stories about the same hypothesis. As hurricanes become more severe in terms of minimum pressure, hurricanes named after men kill more people.
I said before that the authors could have told a “sexism kills” story regardless of whether it was “female hurricanes” or “male hurricanes” that killed more people. It’s way worse than this. The authors could have told the opposite “sexism kills” story — with a statistically significant result either way! — from the same model.
Let’s review the four key analytic steps involved in going from these data to a paper titled “Female hurricanes are deadlier than male hurricanes”:
1. No significant difference in deaths between female hurricanes and male hurricanes. To get a significant difference, you need a model with interaction terms.
2. No significant coefficient for a model that only includes the one interaction term comprising the key result of their paper. To get to a significant result, you need to include a second interaction term, involving minimum pressure.
3. While the interaction effect involving dollar damage is now statistically significant and fits their hypothesis, the other interaction effect involving minimum pressure is also statistically significant but in the opposite direction of you’d expect from their hypothesis.
4. The interaction effect involving dollar damage–the one that says female hurricanes are deadlier–is what they talk about in the paper. It even gets a graph! Meanwhile, the interaction effect involving minimum pressure–the one that says male hurricanes are deadlier–only ends up in the fine print of the methods section and in the online supplement, and it’s presented without any mention of how it has the wrong sign.