Occasionally social scientists become interested in whether Americans are becoming “more polarized” in their opinions. The obvious strategy for considering this question is to take a bunch of survey items that have been asked of comparable samples in the past and now, and to look at whether people hold more divergent views now than they did then. Studies that use this method can be expected to find that American opinion is not becoming more polarized, allowing social scientists to tut-tut journalists and whoever else has been saying otherwise. The only problem is that the research strategy on which this conclusion is based is deeply, perhaps irredeemibly, flawed.
Why? Well, consider the following sentence quoted by Ken over at Marginal Utility:
By 1996 the percentage willing to vote for a black candidate reached 93 percent, so close to unanimity that the survey dropped the question.
Ken mocks the statement because of the difference between 93% and actual unanimity. Something the sentence reveals, though, is that people buying survey time are typically interested in questions that vary. If they are asking a question that doesn’t vary, it’s for some reason, like perhaps because it has been asked repeatedly in the past. There are questions on the General Social Survey, for instance, that have been retained despite not yielding much variance anymore, precisely because they are items about which people did vary in the past. Rewind that and consider again. Way back when, if the issue was one that had as much consensus as it does now, the question never would have been asked in the first place, and it would be unavailable for comparing whether opinion has become more polarized. So, items that would provide evidence of polarization — consensus then, divergence now — are disproportionately less likely to be part of the universe of available items for comparison over time, while items that provide evidence of no polarization — divergence then, consensus now — are disproportionately more likely. And thus researchers claim to producing findings about the world of public opinion when the patterns in their data actually reflect the world of public opinion surveys.
I’m not even sure it’s possible to draw conclusions about whether people are or are not becoming more polarized in general in their opinions, just because what constitutes the universe of opinions over which one is making such a claim seems a deeply murky concept.