my thoughts on that hurricane study

I’ve been active on Twitter regarding the well-publicized study that hurricanes with feminine names kill more people than hurricanes with masculine names. Let me provide an anthology of my thoughts here:

1. The study is about what caused the deaths of real people, and makes claims about how real deaths of real people in real hurricanes could have been prevented. This is serious business.

2. You can download the data yourself following the link in the paper. Everything I looked at you can look at yourself.

3. There is no significant bivariate relationship between the masculinity/femininity of hurricane names and deaths. To get significant results, you have to use a model with interaction effects, meaning that, as storms become more severe, the increase in deaths is bigger for feminine-named hurricanes than masculine-named hurricanes. Unless there was a strong theory behind that particular notion, the lack of a statistically significant bivariate relationship should probably stand as a red flag.

4. Fully replicating the key model of their paper (Table S2, Model 4) is this easy:

. nbreg alldeaths c.ZMasFem##(c.ZMinPress c.ZNDAM)

5. The authors’ comments emphasize how the above code comprises a “sophisticated count model,” instead of the OLS regression that apparently somebody else used in questioning their result. As a practical matter, note that this sophistication in Stata simply involves typing “nbreg” instead of “regress.”

Regarding overall sophistication, to give a simple example, it might seem obvious that an explanatory variable like the dollar value of hurricane damage would fit better if it was logged. A few seconds of work confirms this intuition, but it is unclear whether or how much probing like this was done by the authors.

6. The effect sizes implied by the model are astonishing. The example the authors themselves give in the paper is that if a hurricane named Eloise killed 42 people, the same hurricane named Charley would be predicted to only kill 15. In other words, if true, most of the actual deaths in a severe feminine-named hurricane could have been prevented if only the hurricane had a masculine name.

7. Hurricane Andrew is of particular interest for the model, because it was very severe in terms of damage but not so many people died. 62 people died in Andrew, which their model fits well, predicting 59 deaths. If we pose the counterfactual of what would have happened if the hurricane had been named Diana instead, the model predicts over 25,000 people would have died. In other words, the authors implicitly claim that tens of thousands of Floridians owe their lives to the fact that Andrew was not preceded by another storm that season, because then what we know as Hurricane Andrew would have been called Hurricane Bonnie.

8. The significant coefficients in their key model both become nonsignificant and presumably nonpublishable (p > .25) if we drop only two hurricanes: Andrew and Diane (which killed 200 people in 1955).

9. You can verify #7 and #8 easily yourself. For #8 just add an “if” condition to the above model. For #7, generate a set of predicted values, then switch the MasFem value of Andrew to .98 (that of Diana) and generate predicted values again. Feel free to double-check my work and let me know if you get different results.

10. The authors have issued a statement that argues against some criticisms of their study that others have offered. These are irrelevant to the above observations, as I’m taking everything about the measurement and model specification at their word–my starting point is the model that fully replicates the analyses that they themselves published.

11. A qualification to #11 is that one of their comments is that they deny they are making any claims about the importance of other factors that kill people in hurricanes. But they are. If you claim that 27 out of the 42 deaths in Hurricane Eloise would have been prevented if it was named Hurricane Charley, that is indeed a claim that diminishes the potential importance of other causes of deaths in that hurricane.

12. Much of their paper is actually a series of psych experiments that suggest that people may anticipate more harm to a masculine-named hurricane than a feminine-named hurricane. To me, it is inconceivable that one could take the magnitude of effects reported in these experiments and use them to construct a theory that would predict most lives in a severe feminine-named hurricane could be saved simply by renaming it.

13. Nevertheless, it is plausible to me that people could anticipate a masculine-named hurricane in more severe terms than a feminine-named one, and that sometimes people die as a result. What’s absolutely implausible to me is that this is responsible for a large portion–much less most–of the deaths in severe hurricanes.

14. The preventable disaster here is that the study was dramatically underpowered from the start. To see why, let’s give the authors the benefit of the doubt that their model specification was really both their a priori theory and the correct way to model the data. Then, let’s consider what would minimally be needed for the interaction effects in question to be statistically significant. (In other words, instead of the actual coefficients for the interaction terms, let’s multiply their standard errors by 2 and pretend those were the coefficients.)

My calculations still yield roughly 6500 deaths instead of 60 if Andrew had been called Diana. In other words, this way of testing a hypothesis makes no sense if you think hurricane names might have a real, but modest, effect on how many people die. The test only makes sense if your a priori theory is that hurricane names have a large enough effect to be the major reason people die in severe hurricanes with women’s names.

15. The authors’ university issued a press release with a dramatic presentation of results. The release includes quotes from authors and a photo, as well as a quote from a prominent social psychologist calling the study “proof positive.” So this isn’t something that the media just stumbled across and made viral. My view is that when researchers actively seek media attention for dramatic claims about real deaths, they make their work available for especial scrutiny by others.

16. As a coda that may or may not be relevant to the case at hand, I will confess that I have become especially impatient by the two-step in which a breathless set of claims about findings is provided in a press release, but then the authors backtrack when talking to other scientists about how of course this is just one study and of course more work needs to be done. In particular, I have lost patience with the idea the media are to blame for extreme presentations of scientists’ work, when extreme presentations of the scientists’ work are distributed to the media by the scientists’ employers.

10 Comments

  1. Posted June 3, 2014 at 9:52 pm | Permalink

    From their statement:
    “However, our analysis primarily focused on the femininity-masculinity of names, not only on male/female as a binary category. ”

    In addition to the hurricane death analysis, they did 6 experiments. In 5 of these, the experimental condition was male/female name, like Christopher vs. Christina. In the experiment where this was not the case (Experiment 1), “responses were collapsed, respectively, for five hurricanes with male names (α = 0.571) and five hurricanes with female names (α = 0.638).”

    I rate their statement as mostly false.

    Like

    • Posted June 3, 2014 at 10:06 pm | Permalink

      Agreed. I didn’t even get into the continuous vs. binary part of it. I just did a few little things from the model that they actually published.

      Like

    • Posted June 3, 2014 at 11:16 pm | Permalink

      The whole issue of treating gender as continuous is especially silly given that in the archival study their coding of gender is so bimodal that for all practical purposes it’s a dummy (albeit one taking codes of more or less 2/9 instead of exactly 0/1). I’m highly skeptical that the subtle gradations in femininity between “Ginger” and “Florence” are really doing anything but providing an excuse for including the 1953-1978 period.

      Moreover, I’m not sure how meaningful I find the concept of femininity as rated by nine present-day coders as compared to the expressed behavior of contemporaneous parents. It’s a bit tricky since the hurricanes tend to have diminutives (eg “Alex”) and some of the hurricane names are weird (eg “Easy”) but if you look at Social Security data from the relevant years you tend to see that the names are entirely attached to children of a single gender. For instance, the ratio of male to female babies born in 1953 named “Florence” was 0:1124 and the ratio of male to female babies born in 1989 named “Hugo” was 524:0.* The single most androgynous name according to the nine contemporary coders was “Ione” but parents (and presumably hurricane victims) in 1955 seemed to understand it as completely feminine (0:42), perhaps because knowledge of Greek was more widespread.

      *Technically 0-4 since SSA only reports name-sex-year combinations greater than 5.

      Like

      • Posted June 3, 2014 at 11:29 pm | Permalink

        ps, I chose “Hugo” and “Florence” because these were described as relatively androgynous names in the study but which don’t involve ambiguities of stemming. Notably this leaves out “Alex” which in 2004 had a ton of spelling variations but which overall was in fact an androgynous stem/diminutive.

        Like

      • Posted June 4, 2014 at 12:25 am | Permalink

        Great point. Crazy to use today’s students instead of naming data.

        Like

  2. Posted June 4, 2014 at 12:02 am | Permalink

    I agree, Jeremy. Though, I confess, I felt sorry for them for a couple hours this afternoon, before I saw the press release and some of their comments (in the LA times: “We had a hunch that there would be some gender biases, but we were quite stunned by the degree of this effect.”

    Anyway, here’s my bivariate plot from their data: https://twitter.com/familyunequal/status/473866004041596928

    Like

    • Posted June 4, 2014 at 8:53 pm | Permalink

      I hear you about feeling sorry for the investigators. But, I don’t know, while I’ve always been inclined to give greater value to the “winnowing” function of academic discourse than many colleagues, I think I’m moving even stronger in that direction. Regardless of what gets said on blogs, we’ll see appearances of this study crop up in pop psychology books–many of which will be written by actual academic psychologists–for the next twenty years.

      Granted, this may seem to be a benign fiction, but I’ve come to think these things may actually add up to big public costs in terms of promoting a view of the world in which big effects result from incidental things. But, if I’m going to go on about that coherently, I should probably just make it a post in its own right.

      Like

  3. Posted June 4, 2014 at 2:23 am | Permalink

    Reblogged this on Rene Bekkers and commented:
    Nice discussion of PNAS study on hurricane naming effects.

    Like

  4. Posted June 4, 2014 at 10:31 am | Permalink

    See also this replication (w R code & diagnostic graphs) showing that the correct specification is a quadratic for damage and that when you do so the gender effect drops out

    http://rpubs.com/oharar/19171

    Like

  5. nyoungman
    Posted June 6, 2014 at 6:53 pm | Permalink

    I’m a disaster sociologist but not a statistician, so I’m only following your critique partway here, but one thing that galls me to no end about these experiments is the use of a convenience sample of college students in Illinois who are *highly* unlikely to have any real experience with hurricanes. Try doing this study on the Gulf Coast and let’s see what you get (Betsy! Camille! Katrina! oh my!). It also seems to me that, given how unique and quirky each hurricane is, and how much the storm’s strength, storm surge height, and the population density of the area where it makes landfall matter, the sample size really isn’t big enough here to make meaningful comparisons between male- and female-named storms (whether or not the data from before the used male names is included). Any thoughts on these issues?

    Like

4 Trackbacks

  1. By Hurricanes vs. Himmicanes on June 5, 2014 at 9:25 am

    […] cleanest discussion I’ve seen of this is by Jeremy Freese, here. Freese makes a bunch of detailed points and then gets to […]

    Like

  2. […] The story’s on the sister blog and I quote liberally from Jeremy Freese, who wrote: […]

    Like

  3. By Friday Five: Returns | The Fifth Floor on June 7, 2014 at 12:26 am

    […] And, speaking of strong women, you may have read that female hurricanes cause more deaths because people don’t take them as seriously. Before you believe that you should look more closely at the methodology, which is exactly what they did @ScatterPlot […]

    Like

  4. […] function. It came up recently in some discussions on the scatterplot blog by Jeremy Freese (see 1 & 2) critiquing the PNAS paper on the effect of female named hurricanes on death tolls, so I […]

    Like

Follow

Get every new post delivered to your Inbox.

Join 1,305 other followers

%d bloggers like this: