step #1: repeat

I was re-reading Feynman’s essay on “Cargo-Cult Science,” largely to look at the context of the marvelous quote “The first principle is that you must not fool yourself–and you are the easiest person to fool.” But farther on I saw this part about repeating experiments:

When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this–it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A. So her proposal was to do the experiment under circumstances Y and see if they still did A.

I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person–to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.

She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.

Thinking the way the professor thinks in this story has gotten whole fields into trouble.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

2 thoughts on “step #1: repeat”

  1. If it’s true that social systems are much higher dimensional than those the hard sciences work with, replication (which is really a principle applicable to any method, qualitative or quantitative) becomes even more necessary because the probability that unobserved confounds drive a result is much greater. I think that greater complexity also implies that it is much much harder for us to replicate, and given the enormous costs to any individual in terms of getting a result at all, it’s maybe not surprising that we get an institution that selects, through incentives, for novelty rather than robustness.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s