replicating the future

Gelman post on meta-analysis of the Daryl Bem research on precognition (yes, precognition):

The ESP context makes this all look like a big joke, but the general problem of researchers creating findings out of nothing, that seems to be a big issue in social psychology and other research areas involving noisy measurements. … I have a feeling that the authors of this paper think that if you have a p-value or Bayes factor of 10^-9 then your evidence is pretty definitive, even if some nitpickers can argue on the edges about this or that. But it doesn’t work that way. The garden of forking paths is multiplicative, and with enough options it’s not so hard to multiply up to factors of 10^-9 or whatever. And it’s not like you have to be trying to cheat; you just keep making reasonable choices given the data you see, and you can get there, no problem. Selecting ten-year-old papers and calling them “exact replications” is one way to do it.

I think the parapsychology research is actually extremely useful, especially if one is willing to take as incorrigible the proposition that parapsychological phenomena aren’t real. Because then parapsychology serves as a kind of control group for science practice, and what’s striking about the Bem research is how much it looks like ordinary psychological science–even psychological science that goes above and beyond the norm–and yet the findings are what they are.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

One thought on “replicating the future”

  1. I have a line that I inflict upon my students, grad and undergrad, that relates to this. I think that science is like playing a game of twenty questions with the universe; we ask about things with research, and results are the answers. And we’re lucky because the universe never, ever lies. It always answers our questions and does so perfectly truthfully. The problem is that quite often, it turns out that we didn’t ask the question that we meant to ask, so the answer doesn’t mean what we think it means. And so the art of research isn’t just learning how to ask questions and get answers, but how the ask the RIGHT questions in the RIGHT way. I keep coming back to this with the Bem research: the findings are what they are, but that doesn’t indicate that they mean what we think they mean.


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s