the hurricane name study gets worse

HurricaneStudyKeyTableWithHighlights

Hurricane Name Study, how I wish I could quit you. But an inquiry prompted me to look some more. Again, you can download the data yourself and replicate their key model, Model 4 above, using the half-tweet’s worth of code I included earlier.

The table isn’t in the paper, only their supplemental materials. Notice there are two significant interaction effects: one for the dollar damage of a hurricane (highlighted in green), and the other for the minimum pressure (highlighted in yellow). Both are severity measures. You might think that because both coefficients have the same sign, they both are consistent with the story that, as hurricanes become more severe, the death rate goes up faster for “female hurricanes” than “male hurricanes.”

Hey, wait! Don’t more severe hurricanes have lower minimum pressure? Why, yes. You can confirm this several ways, but notice how the pink coefficient for the main effect of pressure has a negative sign. In other words, the green coefficient and yellow coefficient are actually telling you opposite stories about the same hypothesis. As hurricanes become more severe in terms of minimum pressure, hurricanes named after men kill more people.

I said before that the authors could have told a “sexism kills” story regardless of whether it was “female hurricanes” or “male hurricanes” that killed more people. It’s way worse than this. The authors could have told the opposite “sexism kills” story — with a statistically significant result either way! — from the same model.

Let’s review the four key analytic steps involved in going from these data to a paper titled “Female hurricanes are deadlier than male hurricanes”:

1. No significant difference in deaths between female hurricanes and male hurricanes. To get a significant difference, you need a model with interaction terms.

2. No significant coefficient for a model that only includes the one interaction term comprising the key result of their paper. To get to a significant result, you need to include a second interaction term, involving minimum pressure.

3. While the interaction effect involving dollar damage is now statistically significant and fits their hypothesis, the other interaction effect involving minimum pressure is also statistically significant but in the opposite direction of you’d expect from their hypothesis.

4. The interaction effect involving dollar damage–the one that says female hurricanes are deadlier–is what they talk about in the paper. It even gets a graph! Meanwhile, the interaction effect involving minimum pressure–the one that says male hurricanes are deadlier–only ends up in the fine print of the methods section and in the online supplement, and it’s presented without any mention of how it has the wrong sign.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

4 thoughts on “the hurricane name study gets worse”

  1. Wow. I’m someone who has been on the receiving end of a lot of skepticism about a paper, and even I have basically no sympathy for these guys.

    Like

    1. Yeah, the earlier stuff looked more like they didn’t really know what they were doing. This looks more like they (kind of) knew what they were doing and went ahead with some choices that it’s hard for a bystander to feel good about. Hard for me to understand how they could have seen the other coefficient and not recognized that it was evidence against their hypothesis.

      Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s