theoreticians and the polls

In recent days I’ve chatted with several theory-oriented colleagues (two in English, one in poli sci) about the polling sites. Typically the question is which of the aggregator sites I prefer: Pollster, 538, or Real Clear Politics. (For the record, I typically watch Pollster most carefully.)

What I find interesting is that these colleagues–and, by the way, I, too–buy the critique of public opinion research launched by, e.g., Adorno, Horkheimer, Pollock, Bourdieu, Habermas, etc., that holds polls to be, at best, productive fictions. The performativity claim can get us only so far–but the ontological status of the opinions being “measured” here is hardly clear, and is never addressed by any of the polling sites.

My question, then, is not just why they’re (we’re!) paying attention to the polls. The question is why they/we seem to be doing so in such a rapt fashion, following every known-to-be-meaningless percentage point change with such fascination, even morbid curiosity. This strikes me as an almost Lacanian quest for the true public as an objet petit a–an unattainable object of desire, fetishized and packed into an object of worship, a totem, in the moving polling average.

Author: andrewperrin

University of North Carolina, Chapel Hill

13 thoughts on “theoreticians and the polls”

  1. There is a similar paradox in the entertainment industries, where attributes known to be uncorrelated (or weakly correlated) with sales/viewership are still the focus of a great deal of attention. I don’t have brilliant insights (YET) except to note that fuzzy knowledge is not useless in uncertain (and risky!) situations, and that polls (or press kits, or TV series pitches) do “work” as occupational objects.

    Like

  2. Can you explain more the criticism of public opinion research from Adorno, etc.? My initial response is that this suggests how pedantic arguments from these lit-crit type theorists can be. In election polling, it is clear that the pollsters care mostly about predictive validity, which of course is assessed most clearly by comparing poll results from the day before the election to the election voting outcomes.

    Like

  3. I’m a believer in polls, long as you internalize all the caveats behind them. Also, I’m philosophically Bayesian. I have priors and polls are informative sources for updating, even if they have some problems.

    Also, we tend to remember the worst polls rather than the average. Polls are usually in the ball park, if they are done well. For every Obama/New Hampshire screw up, we’ve got tons of polls that were kind of right. Heck, in the primary 2008 season, the rolling averages were usually spot on, except New Hampshire, which was a huge mess up. And remember – the polls got the GOP contest correct in that state.

    If that ain’t good enough, then these people are not in my reality based community!

    Like

  4. I found this post bizarre.

    What do you mean that polls are “productive fictions?” Methodologically sophisticated observers (e.g. Gary King, Andrew Gelman, and many others etc.) wouldn’t take polls at face value either. The validity of any kind of poll is periodically reevaluated in journals like Public Opinion Quarterly, but usually in concrete ways, and with suggestions for improvement, rather than wholesale dismissal via jargon (“productive fiction.”)

    “The question is why they/we seem to be doing so in such a rapt fashion, following every known-to-be-meaningless percentage point change with such fascination, even morbid curiosity.”

    Who is following the polls with rapt attention? (You might actually need a poll result to find out.) It makes it more exciting to emphasize the competitive nature of the election contest, and treating the noise in polling results like gains and losses in a horse race does that.

    The poll is like an object of worship, a totem? To whom?

    If the people you are talking about who are watching the polls with rapt attention are most people, the maybe this is a case of typical intellectual confusion about non-intellectuals.

    Like

  5. I think it’s not very useful to just say that these measures are bad but to keep in mind the ways in which the “problems” are really “data” for the specific context you’re trying to understand. If the context you’re predicting is weird in similar ways to the weirdness of the instrument then the weirdness is a good thing (for that specific purpose).

    So for instance the kind of questions that tend to have Likert scales attached to them are problematic in all sorts of ways (especially salience) and may not map particularly well onto behavior or even less abstract attitudes. In the abstract, you could say that “Do you plan to vote? If so, for McCain or Obama?” is also flawed in some way, but if you’re trying to predict how a person will behave in the equally artificial and dichotomous context of the voting booth, it’s pretty good. On the other hand this same question might be less useful for other purposes, such as trying to measure the ideological skew of various occupations. Likewise, pollsters are treading on thin ice when they abandon the reasonably valid context of trying to project votes in a highly salient election and use a lot of vague likert-scaled questions for a factor analysis that slices us into “national security moms” or “enterpisers.”

    I don’t know that much about about the technical debates of psychometrics, but my understanding is that a lot of what IQ tests measure (and particularly what the Flynn effect is picking up) is the ability to process highly stylized and decontextualized puzzles — a learned and very culturally specific cognitive style. Such a measure may be only a mediocre predictor of how someone will solve the vast majority of fuzzy and messy problems we encounter in the real-world but it seems like it ought to be a pretty good predictor of how someone will behave in situations that are about as stylized and decontextualized as the testing instrument itself. So for instance, I would expect the GRE to be better at identifying promising mathematicians than the ASVAB is at identifying promising non-commissioned officers for the marines or the GMAT at picking promising business executives.

    Like

  6. Wow, I didn’t expect to get quite the degree of reaction I did! Since I’m in a psychoanalytic mode I’m almost inclined to chalk it up as a reaction formation. Meanwhile, though, some specific thoughts:

    @2.lostmyedgelongago: the critiques launched by these very diverse theorists share a number of themes. A few of them:

    The constraint critique: what’s going on in respondents’ heads is distorted by the fixed-answer constraints of polls;
    The performativity critique: respondents learn from polls to behave in the ways pollsters expect;
    The shallowness critique: polls just don’t represent much of the complex thinking of respondents;
    The publicness critique: polls collect and aggregate private opinions, sidestepping the centrality of the public in public opinion;
    The stable individual critique: polls sample individual people, not contexts or groups, so they assume opinions to be (relatively) stable properties of individuals; and
    The presentation of self critique: respondents have what Chris Smith has called “Second-Order Desires About Desires” (SODADs), and seek to present strategically designed facades instead of “true” selves. (Pollsters call this “social desirability bias.”)

    I agree that pollsters are usually interested in predictive validity, but at the same time the language they use is about the immediate: polls “move,” the public “reacts,” etc., which are about seeking to represent an unknown, but assumed present, underlying public at the time of the poll, not in prediction of its eventual behavior.

    @3.fabio: I agree with what you said; it’s not a matter of “good enough,” but rather of ontological status.

    @4.daedalus: Get thee to Sarah Igo. Sorry you found it bizarre. By “productive fictions” I mean that they are creative, even aesthetic, products, that seek to represent an assumed, but inaccessible, public opinion. What it takes to understand them is not “methodological… sophistication” but theoretical sophistication. If you read my original post carefully, you will note that it was four theoreticians who follow the polls with such rapt attention: myself, a political theorist, and two English lit-theorist types. Of course it “makes it more exciting.” But that’s an aesthetic criterion, not an empirical one. I think the poll is like a totem — in a Freud/Durkheim sense — because it commands a kind of ritual respect beyond its material reality. Finally: again, if you would read the post carefully before responding, you would avoid such nonsensical claptrap as “typical intellectual confusion about non-intellectuals.”

    @5.gabrielrossman: I don’t think I said “bad,” I said “productive fiction.” I like fiction, don’t you? And I sure like productive. And of course the weirdnesses and error terms are data!

    Like

  7. Fabio, it’s hilarious that you say that, because I said almost the same thing to Steve Vaisey one time in a theory seminar. He pointed out that dictionary.com is your friend. Essentially, I take ontology to mean what’s really there, as opposed to what’s observed. So unlike, say, public health or political science, whose interests are essentially phenomenological — that is, they are interested in happenings, not essences — sociology (sometimes) is interested in ontology.

    For the purposes of this discussion, what I mean by this is that I’m interested in the ontological claims made or assumptions made by public opinion research: what is the Thing being measured here?

    I think your position (stop worrying about ontology and learn to love the bomb) is a common one, but not a terribly satisfying one on a theoretical level, IMHO.

    Like

  8. I personally don’t find much value in these critiques as stated in @6.
    Constraint: doesn’t the fixed-category answers on political polls just reflect how choices appear in the voting booth?
    Shallowness: seems obvious in this case, horse race polls focus on who you will vote for, not why.
    Publicness: vote counts aggregate individual votes to get percentages, so aggregating individual opinions seems like the exactly right way to go here.
    Stableness: no, they don’t assume attitudes are stable, that’s a reason why we get new polls every day.
    Presentation of self: I get this, but I like “social desirability bias” (or “Bradley effect”) better.
    To the extent these are issues, it seems like survey-research types are well aware of them and often have done specific research studies on these topics.
    On the immediate use of pre-election polls, it seems like the polls try to address a fairly clear counterfactual: they’re trying to predict how, if a vote were suddenly held that day, the vote would turn out.

    Like

  9. @10.lost: I agree with your comments on constraint and shallowness if you restrict discussion to just the horse-race polls predicting election outcomes. But most public opinion research goes well beyond that, dividing voters into types, ascribing attitudes and emotions to “The Public,” and in other ways referring to an assumed, but unmeasured, collectivity.

    As for publicness: again, if your goal is to mimic the election, there’s no problem. If it’s to model a public, the problem is quite substantial, as there’s no reason to assume that a public is just the additive aggregation of innumerate privates.

    Taking multiple polls only corrects for instability over time, not across contexts.

    Like

  10. Andrew Perrin: Thank you for the reference to Sarah Igo. That seems to be very interesting book, reading the first chapter which was posted online.

    “If you read my original post carefully, you will note that it was four theoreticians who follow the polls with such rapt attention: myself, a political theorist, and two English lit-theorist types. Of course it “makes it more exciting.””

    I didn’t think you were referring to just the four of you, but rather making a generalization. I don’t think you would have thought it important to write about here if thought it really just referred to four people.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.