I’ve been active on Twitter regarding the well-publicized study that hurricanes with feminine names kill more people than hurricanes with masculine names. Let me provide an anthology of my thoughts here:
1. The study is about what caused the deaths of real people, and makes claims about how real deaths of real people in real hurricanes could have been prevented. This is serious business.
2. You can download the data yourself following the link in the paper. Everything I looked at you can look at yourself.
3. There is no significant bivariate relationship between the masculinity/femininity of hurricane names and deaths. To get significant results, you have to use a model with interaction effects, meaning that, as storms become more severe, the increase in deaths is bigger for feminine-named hurricanes than masculine-named hurricanes. Unless there was a strong theory behind that particular notion, the lack of a statistically significant bivariate relationship should probably stand as a red flag.
4. Fully replicating the key model of their paper (Table S2, Model 4) is this easy:
. nbreg alldeaths c.ZMasFem##(c.ZMinPress c.ZNDAM)
5. The authors’ comments emphasize how the above code comprises a “sophisticated count model,” instead of the OLS regression that apparently somebody else used in questioning their result. As a practical matter, note that this sophistication in Stata simply involves typing “nbreg” instead of “regress.”
Regarding overall sophistication, to give a simple example, it might seem obvious that an explanatory variable like the dollar value of hurricane damage would fit better if it was logged. A few seconds of work confirms this intuition, but it is unclear whether or how much probing like this was done by the authors.
6. The effect sizes implied by the model are astonishing. The example the authors themselves give in the paper is that if a hurricane named Eloise killed 42 people, the same hurricane named Charley would be predicted to only kill 15. In other words, if true, most of the actual deaths in a severe feminine-named hurricane could have been prevented if only the hurricane had a masculine name.
7. Hurricane Andrew is of particular interest for the model, because it was very severe in terms of damage but not so many people died. 62 people died in Andrew, which their model fits well, predicting 59 deaths. If we pose the counterfactual of what would have happened if the hurricane had been named Diana instead, the model predicts over 25,000 people would have died. In other words, the authors implicitly claim that tens of thousands of Floridians owe their lives to the fact that Andrew was not preceded by another storm that season, because then what we know as Hurricane Andrew would have been called Hurricane Bonnie.
8. The significant coefficients in their key model both become nonsignificant and presumably nonpublishable (p > .25) if we drop only two hurricanes: Andrew and Diane (which killed 200 people in 1955).
9. You can verify #7 and #8 easily yourself. For #8 just add an “if” condition to the above model. For #7, generate a set of predicted values, then switch the MasFem value of Andrew to .98 (that of Diana) and generate predicted values again. Feel free to double-check my work and let me know if you get different results.
10. The authors have issued a statement that argues against some criticisms of their study that others have offered. These are irrelevant to the above observations, as I’m taking everything about the measurement and model specification at their word–my starting point is the model that fully replicates the analyses that they themselves published.
11. A qualification to #11 is that one of their comments is that they deny they are making any claims about the importance of other factors that kill people in hurricanes. But they are. If you claim that 27 out of the 42 deaths in Hurricane Eloise would have been prevented if it was named Hurricane Charley, that is indeed a claim that diminishes the potential importance of other causes of deaths in that hurricane.
12. Much of their paper is actually a series of psych experiments that suggest that people may anticipate more harm to a masculine-named hurricane than a feminine-named hurricane. To me, it is inconceivable that one could take the magnitude of effects reported in these experiments and use them to construct a theory that would predict most lives in a severe feminine-named hurricane could be saved simply by renaming it.
13. Nevertheless, it is plausible to me that people could anticipate a masculine-named hurricane in more severe terms than a feminine-named one, and that sometimes people die as a result. What’s absolutely implausible to me is that this is responsible for a large portion–much less most–of the deaths in severe hurricanes.
14. The preventable disaster here is that the study was dramatically underpowered from the start. To see why, let’s give the authors the benefit of the doubt that their model specification was really both their a priori theory and the correct way to model the data. Then, let’s consider what would minimally be needed for the interaction effects in question to be statistically significant. (In other words, instead of the actual coefficients for the interaction terms, let’s multiply their standard errors by 2 and pretend those were the coefficients.)
My calculations still yield roughly 6500 deaths instead of 60 if Andrew had been called Diana. In other words, this way of testing a hypothesis makes no sense if you think hurricane names might have a real, but modest, effect on how many people die. The test only makes sense if your a priori theory is that hurricane names have a large enough effect to be the major reason people die in severe hurricanes with women’s names.
15. The authors’ university issued a press release with a dramatic presentation of results. The release includes quotes from authors and a photo, as well as a quote from a prominent social psychologist calling the study “proof positive.” So this isn’t something that the media just stumbled across and made viral. My view is that when researchers actively seek media attention for dramatic claims about real deaths, they make their work available for especial scrutiny by others.
16. As a coda that may or may not be relevant to the case at hand, I will confess that I have become especially impatient by the two-step in which a breathless set of claims about findings is provided in a press release, but then the authors backtrack when talking to other scientists about how of course this is just one study and of course more work needs to be done. In particular, I have lost patience with the idea the media are to blame for extreme presentations of scientists’ work, when extreme presentations of the scientists’ work are distributed to the media by the scientists’ employers.