biernacki, “reinventing evidence”

OrgTheory’s current book forum is on Richard Biernacki‘s Reinventing Evidence in Social Inquiry. I provide my views here to contribute to the discussion.

Biernacki attempts a wholesale indictment of the practice of “coding” texts as a social scientific technique. Through careful attempts to replicate three studies, Biernacki seeks to show that the attempt to bridge interpretive and analytical sociology by sampling and categorizing bits of text is “unfeasible.” Essentially, I believe he hopes to demonstrate a kind of methodological “non-overlapping magisteria” claim: that interpretive approaches are sui generis and uniquely capable of successfully comprehending textual and cultural evidence, and analytical techniques are epistemologically bankrupt. He does so by a cherished if underused scientific technique: replication, in this case of three important works in cultural sociology. The works are Bearman and Stovel’s “Becoming a Nazi: A Model for Narrative Networks” (Poetics, March 2000); John Evans’s 2001 book, Playing God? , on which Biernacki has already commented extensively and very similarly; and Wendy Griswold’s 1987 “The Fabrication of Meaning”.

I say “I believe” that is the point of the book, because unlike his prior book (The Fabrication of Labor, a magnificent historical study demonstrating the independent effect of national culture on early modern economic organization in England and Germany) the argument in Reinventing is hidden behind a smokescreen of arrogant posturing, making it difficult to evaluate the underlying idea and its defense.

In short, while there are some apt points in the book, in general it is pompous in style, muddled in evidence, vastly overstated in scope, mean-spirited in approach, and epistemologically indefensible.

I begin by identifying and evaluating four points that run through Biernacki’s argument. I then demonstrate the grounds for each of my charges above. I close with some thoughts on the book as a whole and the controversy surrounding its publication.

Point 1: Textual wholism. This claim runs throughout the book, and is the most consistent ontological claim. Here’s one example, from page 11:

 [Using] ritual productions…. you can rip everyday implements or emblems out of their original contexts and find new ways for them to work as “symbols” in the cleansed space of the performance. Yesterday’s wine turns to Christ’s blood at the Last Supper or Halloween cloaks into Shakespearean robes. The coding of texts works much the same way, because the operation assumes one can convert regenerating meanings into an isolated token, a datum label. It dissolves an opus’s multilayered architecture and resonances into a discrete, single-level cipher (or linear string of ciphers) that can be brought into previously unimaginable relations to each other.

By page 16, it has become a programme:

that we not grant epistemological legitimacy to the interpretive model with which we start; that there is no pigeon-holing of textual evidence…; meaning is bound to concrete, unsystematizable prototypes; and, finally, that we read the texts as they are embedded in a culture different from the one by which we create our ideal models as investigators….[in order to] guard against over-rationalizing methods that confuse investigators’ analytic categories with those naturally “in” the evidence.

And later: “Dramatic omission and suggestive possibility in narratives stimulate our engagement as readers but render consistent coding nearly impossible” (37).

There are two pieces to textual wholism as presented here. The first is internal: the meanings of words and phrases are wholly dependent upon the remainder of the text in which they are set, so analyzing them outside that context evacuates them of meaning; “fancifully expung[ing] from view the multiple and idiosyncratic communicative projects inside of which each text feature was originally configured to function” (24). The second is external: texts themselves are set within historical and cultural contexts, so they must be read “as they are embedded in a culture different from” the analyst’s.

Both of these claims are absurd. The former is absurd because it assumes an untenable theory of language. In order for the claim to be correct, it must be the case that words and phrases have no meaning outside their textual context. But that is simply false; while the specific meanings are conditioned, of course, on context and sequencing, words and phrases also reference meanings from outside a given text. Indeed, that’s probably the most important source of meaning in written language! The latter claim is absurd for reasons that ought to be obvious: the claim that texts can only be interpreted within their “own” cultures would leave all texts effectively without meaning, since one of the core properties of texts is that they are portable across space and time. Any interpretive act involves reinterpreting a text in a new context: that of the reader. Hence it is fatuous to claim that there are some categories “naturally ‘in’ the evidence” and others imposed from without. Indeed, Biernacki ought to know as much, since the genius of The Fabrication of Labor emerges precisely from the practice of reading texts from outside their “own” cultures to bring them into comparative relief.

Point 2: Scholars ought to document and publish their coding assignments and decisions.

“Everyone,” writes Biernacki with his characteristic overreach, “(researcher, reader, dean) insulates what transpires in coding as an unapproachable black box. Investigators vouch for their ‘data’ but do not necessarily release information about the sample sources, let alone the goings-on by which they translated texts into outputted variables” (19). With respect to Bearman and Stovel, he writes: “The ‘data’ are interpretive guesses all the way down, with no bottoming out in ascertainable givens” (36).

This point is technical and pragmatic, not epistemological. Ironically, it is a point far more comfortable in the realm of social science (the latter word always enclosed in scare quotes here, as if the very mention of science is taboo) than in Biernacki’s preferred hermeneutic approach. Yes, scholars should make their data available and their procedures transparent. It appears from Biernacki’s account that these three scholars did not do so particularly well, though it bears mentioning that between 11 and 25 years passed between the works’ publication and Biernacki’s replication.

Professors Bearman and Griswold undertook to locate their coding records, and they were patient in communicating about their efforts…. Transparency of data in a research community is championed by journals and by granting agencies. In the cross-genre of qualitative coding, this norm has become largely inoperable.

So says he – with N=3, and after considerable time had passed. (Bearman wrote me that “The box with the stories really did become a home for squirrels.”)

I doubt if anyone would disagree that transparency would be better. But there’s nothing inherent to coding that makes transparency impossible. In fact, computer technology and the price of storage have improved enough to make it easier since the studies Biernacki critiques. Biernacki inflates what are at worst sloppy practices into methodological crimes.

Point 3: Studies using coding are presented as authoritative.

“’Becoming a Nazi’ stages the discovery of network theory as the model of intelligibility for all facets of social experience,” writes Biernacki (56), with a footnote that is nearly totally irrelevant to the statement (it refers to p. 51 of Algirdas Julien Greimas’s The Social Sciences: A Semiotic View. I can find nothing on that page that is relevant to Biernacki’s claim, but I’ve posted the page here for anybody who wants to look.)

But, in fact, the article does no such thing. Rather (from the abstract):

By representing complex event sequences as networks, inducing ‘narrative networks’, it is possible to observe and measure new structural features of narratives…. The substantive idea that we develop in this article is that the observable narrative structure of life stories can provide insight into the process of identity formation.

Or, as Bearman wrote me directly, “The article is very clear about what it is — a picaresque journey.”

Biernacki:

I have mocked the pretensions of a cross-dressing social ‘science,’ not those of conventional natual science, whose clothing sociologists try to wear as their own by presenting ‘large-N’ coding results…. Certainly it [the inevitable handpicking of sundry sources] is more honest than reifying a mechanical en masse choice as natural. (154-155)

This is a straw man, as I’ve demonstrated in the case of Bearman and Stovel above. None of the three works in question actually presents itself as definitive or scientifically totalizing. On this point, Biernacki’s breathless accusation and huffy superiority strongly suggests an inferiority complex on the part of the critic.

Point 4: Each of the three studies made problematic interpretive choices.

Within each replication, there are numerous examples of problems Biernacki locates with the sampling or analytic decisions. One example among many: “For all we know, had Bearman and Stovel calculated the role of anti-Semitism in comparison to variables defined more concretely, anti-Semitism might pop out as ‘high’ in power centrality” (44). There is an extended discussion of selectivity in quote selection by Griswold, who “expounded on how several reviews treated the Trumper ‘scene’ in a novel, whereas most of her preferred examples barely alluded to it” (128).

Biernacki’s criticisms here seem generally believable, but given the overall hostile tone of the book and the fact that there is no response from the authors, I’d tend to withhold judgment on the specifics. Evans has provided a comprehensive, and IMHO convincing, response in the prior book.

The important point here, though, is that Biernacki’s critique of the analytic decisions presupposes that there is an appropriate and valid way of coding texts for analysis. If there weren’t, there would be no adequate grounds for launching the critiques! To the extent that this point is correct, point 1 cannot be, and vice versa.

–*–

Now let me turn to each of the problems I see with Renventing Evidence.

  1. It is pompous in style. Particularly in his extended, disingenuous re-casting of coding-based research as “ritual,” it is consistently dismissive and holier-than-thou. “Only in liturgy,” sniffs Biernacki as one of countless examples, “is the truth of expression decided in such a fashion, by whether the outside world fits the ritual, not by whether the ritual fits the outside world” (69). Or: “Brought to life in social ‘science’ as ritual, familiar instruments of explanation act as if possessed by foreign spirits” (99). “The first step in critical diagnosing of a coding diagram may be insisting on sheer absence of sense” (125). Dan Hirschman is charitable in describing the prose as “dense,” which it is not particularly. Actually it’s quite readable. What it is is pompous.
  2. Its evidence is muddled. Through meticulous attempts at replication, Biernacki describes flaws in the execution of three coding-based studies. Based on these critiques—which go unanswered by the studies’ authors, so we have no adequate way of adjudicating them—he seeks to establish an epistemological dismissal of coding as a theoretical practice. But not only is this dismissal unwarranted by the evidence, the very structure of the critiques themselves actually demonstrates why the dismissal is incorrect.
  3. It is vastly overstated in scope. The title of the book is “Reinventing Evidence in Social Inquiry.” One might think, then, that the book would have something—anything—to say about evidence in some other area of social inquiry. But not so. According to JSTor Data for Research, the word “coding” appears in about 2.7% of sociology articles in 1987 (when Griswold’s article appeared), climbing to 4.7% in 2001 when Evans’s book came out, and to a high of 7.3% in 2011, the last year for which full data are available. Providing a half-baked critique of three relatively old exemplars of a style of sociology comprising well under 10% of the field is hardly reinventing evidence in social inquiry.
  4. It is mean-spirited in approach. Each of the three studies reviewed is treated as utterly without merit, its author(s) apparently ignorant or worse. I assume there is some back-story explaining Biernacki’s antipathy to his colleague and department chair, John Evans, and of course the infamous dean’s warning not to publish the book is wrong on its face. But whatever its source, that antipathy has now led to two extended debates and a scorched-earth style here without any meaningful progress
  5. Finally, it is epistemologically indefensible. The main epistemological claim is wholism: that decontextualized fragments of text have no meaning separate from the surrounding text, and that in turn texts have no meaning separate from the cultural milieux in which they were produced.
  • On the first: “The ‘representative sample’ approach of science cannot accommodate the pursuit of exchanges of cultural significance” (62). “Word frequencies never have a legible relation to cultural competencies, repertoires, or semiotic systems. Numbers do not sketch anything of verbal significance” (143). This is a bizarre theory of language, utterly irreconcilable with any reasonable understanding of how meaning is encapsulated in and extracted from text. Taking it to its logical conclusion, it is an indictment of any kind of analysis (the root, ana-, referring to dismantling an object of interest to understand its parts). All science is about discarding some information in order better to understand the information that remains.
  • On the second: the requirement that texts be interpreted in their own cultures would reduce the meaning in such texts to nothing—the whole point of written text is that it is transportable across time and space and can be interpreted elsewhere in ways that bridge author, reader, and the respective cultural environments of each. The ridiculous requirement that texts be read within their “own” cultures quickly reduces to Borges’ fable of the map.

An arrogant exercise in puffery, Reinventing Evidence offers precious little of value for understanding or reforming social inquiry of any stripe.

6 Comments

  1. Posted April 1, 2013 at 10:57 pm | Permalink

    1. I stand by my claim that the prose is dense (while not disagreeing that it can also be pompous). But perhaps this reflects a different training – I have very little background in cultural sociology, and so was entering many of these debates a bit unprepared. Your first quote provides a nice example:
    “The coding of texts works much the same way, because the operation assumes one can convert regenerating meanings into an isolated token, a datum label. It dissolves an opus’s multilayered architecture and resonances into a discrete, single-level cipher (or linear string of ciphers) that can be brought into previously unimaginable relations to each other.”
    This is chewy text, albeit not indecipherable.

    2. I agree that N=3 is a bit of a problem for the book in terms of its claims about non-reproducibility. That said, Biernacki does go to some length to motivate his case selection on the grounds that these works were especially well-regarded, award-winning, paradigmatic, etc. In regards to the age of the cases… it’s worth remembering that the publication of the book was delayed a fair bit by all of the controversy. Not delayed 11 years of course, but delayed more than one. Also, given the problems with the reproducibility of *quantitative* studies (which should, in theory, be even easier to document and reproduce), I’m willing to believe that this problem will generalize (though, as you say, it’s nowhere near an inherent problem, rather a pragmatic one).

    3. I think there are some good general principles about the dangers of selecting a sample for quantitative coding of texts that automagically generates the results you expected, but I agree that his critique of the very possibility of learning something useful from the process is probably untenable, and that the language of the overall critique will probably prevent most readers from getting much from some of the sub-arguments (such as the discussion of selection bias). The general problem of sampling discourse in such a way as to make claims about that discourse across discontinuities in the field is a messy one, and I found Biernacki’s claims about the particular works studied compelling on this score, and useful for thinking through my own attempts to do similar research in the context of the emergence of macroeconomics.

    4. Lovely Borges reference.

    Like

  2. Posted April 2, 2013 at 12:33 am | Permalink

    Are you going to pretend you don’t know the story behind why Biernacki’s book was banned for so many years? You call him arrogant and pompous, but what of Evans who actively tried to prevent the book from coming out? Deans don’t just pick up a sociology manuscript out of the blue. This is why context matters!

    But seriously, aren’t you and John Evans friends?

    Those needed disclosures aside, did you really just try to dismiss the merits of this book by saying it only addresses a small niche in the field and then search for the word “coding” to see how many articles used that word? Did you even read this book? What about authors that say “thematizing” instead? Further, do you not understand that the book’s title references the ‘reinvention’ done by his case studies, not just himself? Come on!

    Ironically, you argue the same point as Biernacki in this comment you make to Evans on the SSRC blog:

    “Another route involves assuming religious commonality without explicitly offering it as a reason. This arises in all sorts of pseudoreligious controversies such as sex education and the Park 51 controversy. A prime example here is Christine O’Donnell’s justification for her anti-masturbation stance, which is prima facie idiotic: “…if he already knows what pleases him, and he can please himself, then why am I in the picture?” Leaving aside O’Donnell in specific, everybody of course already knows why a real, live sexual partner is in the picture! The statement only makes any sense at all if uttered in a religious context, i.e., one in which there is an assumed religious commonality between the speaker and the audience. In this case, the commonality is the religious assumption that the purpose of sexuality is essentially religious. Thus this deserves to be understood as religious reason-giving even though there is no religious language in the reason!

    My overall point is that religious thinking infuses so much public debate that I’m wary of looking only at explicit reason-giving. I do understand the rationale for doing so in terms of normative deliberative theory, but I’m not sure how far it takes us in terms of the empirical investigation.”

    Here, you are saying that context matters when interpreting someone’s statements. Moreover what matters especially is the audience they are speaking too- you know- the culture.

    Like

    • Posted April 2, 2013 at 1:22 pm | Permalink
      • Yes, John and I are friendly, but I have never spoken to him about this controversy, this book, or his actions in this case. My review is of the book, not of the controversy. Ultimately the book’s importance is as a book, not as an artifact of an intradepartmental fight. As I stated in my review, it strikes me as self-evidently wrong to seek to suppress publication.
      • I don’t call Biernacki arrogant and pompous — having never met him, I couldn’t sustain such a case. The book certainly is both.
      • I dismissed the book’s vast overreach — a quantitative claim addressing “social inquiry” in general — using a quantitative method, yes. I dismissed its epistemological claims using different argumentative approaches.
      • Yes, I read the book, in its entirety, and carefully.
      • My point on the SSRC blog was, as you nicely summarize, “context matters.” That is a trivial point, almost uncontroversial. It is also entirely inadequate as a foundation for the case Biernacki seeks to mount. Incorporating context and audience is entirely possible through methodologies employing coding; indeed, coding could potentially make these tasks easier (see, for example, this article). “Context matters” is worth fifteen minutes in a methods class, not an entire book about all of social inquiry.

      Like

  3. Posted April 2, 2013 at 12:37 am | Permalink

    Oh Wait! I just realized this is April Fools Day! Jokes on me! Carry on.

    Like

  4. Posted April 2, 2013 at 3:39 am | Permalink

    Nice review, Andy. I just finished it, and am not quite as disapproving. Beneath the bluster, Biernacki does raise some valid criticisms. Like all methods, “coding” involves black-boxing lots of stuff, and it doesn’t hurt to peer inside every now and then.

    That being said, I do wonder how we are to grade students’ papers. If I read one and think, “this is an 85%,”am I rendering unspeakable violence upon the text?

    And what about using Rotten Tomatoes to help decide what movie to see tonight? I really like Rotten Tomatoes. A world without it isn’t a world I wanna live in.

    Like

  5. kenkolb
    Posted April 2, 2013 at 1:59 pm | Permalink

    Thoughtful review. Although I typically defend the practice, this recent piece on NPR caught my ear as an example of coding run amok

    http://www.npr.org/blogs/health/2013/04/01/175584297/mining-books-to-map-emotions-through-a-century

    Like

9 Trackbacks

  1. [...] Read Andrew Perrin’s review at Scatterplot. [...]

    Like

  2. By my $0.02 | re-musing on April 2, 2013 at 2:44 pm

    [...] overall reaction isn’t that dissimilar from Andrew Perrin’s, in that i think there are some interesting points to be made in this book, and that we would do [...]

    Like

  3. [...] Since I’ll be travelling this week, I will take a break from the blog. Next Monday, I will resume blogging with Part 2 of our discussion of Reinventing Evidence. (Part 1, book review at Scatterplot.) [...]

    Like

  4. [...] Part 1, Scatterplot review by Andrew Perrin. [...]

    Like

  5. [...] to give less ground, because it is more dismissive and antagonistic in spirit. Andrew Perrin, at Scatterplot, concluded rather harshly (especially compared to Rojas) [...]

    Like

  6. [...] Part 1, Part2,  Scatterplot review by Andrew Perrin. [...]

    Like

  7. [...] has limits.” For example, my previous blog post quoted Perrin’s original Scatterplot post granting a lot of ground on the question of coding’s [...]

    Like

  8. [...] y en particular una reseña del libro en el blog Scatterplot de Andrew Perrin (aquí el link). La tesis del libro es, si se reduce a lo mínimo, que los recientes usos de análisis numéricos [...]

    Like

  9. By Is Social Science a Joke? - Waggish on May 9, 2013 at 4:09 pm

    […] establishment is having an easier time attacking the second thesis, however, judging by Andrew Perrin’s nasty review. Perrin adopts a ridiculous “They aren’t trying to be scientific” defense, which […]

    Like

Follow

Get every new post delivered to your Inbox.

Join 1,127 other followers

%d bloggers like this: