the bigfoot-black swan continuum of behavioral science

Jason Mitchell uses the example of “black swans” to argue that there is a fundamental asymmetry between positive and negative findings in psychology experiments, such that positive findings are the only meaningful findings and negative findings should not be published. The idea is that no matter how many white swans you observe, you don’t know if black swans exist; whereas if you observe one black swan, you know they do (full quote at bottom).

The problem: findings from behavioral science experiments aren’t like being able to hold a black swan by the neck and shout to everyone, “See! I told you they existed!”

Instead, you are presented with papers in which you have to trust researcher reports of what they did to produce a finding that an observed swan was darker than would be expected under the white-swan null (p < .05).

In this respect, positive experimental findings are somewhere on a continuum between Bigfoot and black swans.

There’s all sorts of positive evidence about Bigfoot’s existence, but what does not exist is inarguably decisive evidence: “Mom! Can we go to the zoo and feed the bigfoot?” Whatever the rate that new smudged Bigfoot photos appear in the cryptozoological literature, the continued elusiveness of a decisive demonstration weighs more and more as negative evidence that Bigfoot doesn’t exist the longer it goes on.

When evidence is statistical and demands audience trust, you do not have a black swan in your hand to wave at skeptics. As long as there is this possibility for doubt-in-the-first-place, both positive and negative evidence obviously remain in place for someone trying to figure out what to believe.

Mitchell: “Suppose I assert the existence of some phenomenon, and you deny it; for example, I claim that some non-white swans exist, and you claim that none do … Whatever our a priori beliefs about the phenomenon, from an inductive standpoint, your negative claim (of nonexistence) is infinitely more tenuous than mine. A single positive example is sufficient to falsify the assertion that something does not exist; one colorful swan is all it takes to rule out the impossibility that swans come in more than one color. In contrast, negative examples can never establish the nonexistence of a phenomenon, because the next instance might always turn up a counterexample. Prior to the turn of the 17th century, Europeans did indeed assume that all swans were white. When European explorers observed black swans in Australia, this negative belief was instantly and permanently confuted. Note the striking asymmetry here: a single positive finding (of a non-white swan) had more evidentiary value than millennia of negative observations. What more, it is clear that the null claim cannot be reinstated by additional negative observations: rounding up trumpet after trumpet of white swans does not rescue the claim that no non-white swans exists.”

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

2 thoughts on “the bigfoot-black swan continuum of behavioral science”

  1. Discovering a black swan does not answer the next question of how many black swans exist. Andrew Gelman appears to be correct that true null hypotheses are rare in social science: http://andrewgelman.com/2004/12/29/type_1_type_2_t/. In terms of the effect of X on Y, the null hypothesis claims that X never — in no place — at no time — on no observation — has ever had or will ever have a non-zero effect on Y; that does not seem like an interesting or important claim to investigate in many cases.

    So while Mitchell’s assertion that “a single positive finding…had more evidentiary value than millennia of negative observations” is correct, it applies only if the research question concerns the *existence* of something. However, if we are interested in frequency or effect size, then each independent observation from the “millennia of negative observations” has evidentiary value equal to the “single positive finding.”

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s