biernacki, “reinventing evidence”

OrgTheory’s current book forum is on Richard Biernacki‘s Reinventing Evidence in Social Inquiry. I provide my views here to contribute to the discussion.

Biernacki attempts a wholesale indictment of the practice of “coding” texts as a social scientific technique. Through careful attempts to replicate three studies, Biernacki seeks to show that the attempt to bridge interpretive and analytical sociology by sampling and categorizing bits of text is “unfeasible.” Essentially, I believe he hopes to demonstrate a kind of methodological “non-overlapping magisteria” claim: that interpretive approaches are sui generis and uniquely capable of successfully comprehending textual and cultural evidence, and analytical techniques are epistemologically bankrupt. He does so by a cherished if underused scientific technique: replication, in this case of three important works in cultural sociology. The works are Bearman and Stovel’s “Becoming a Nazi: A Model for Narrative Networks” (Poetics, March 2000); John Evans’s 2001 book, Playing God? , on which Biernacki has already commented extensively and very similarly; and Wendy Griswold’s 1987 “The Fabrication of Meaning”.

I say “I believe” that is the point of the book, because unlike his prior book (The Fabrication of Labor, a magnificent historical study demonstrating the independent effect of national culture on early modern economic organization in England and Germany) the argument in Reinventing is hidden behind a smokescreen of arrogant posturing, making it difficult to evaluate the underlying idea and its defense.

In short, while there are some apt points in the book, in general it is pompous in style, muddled in evidence, vastly overstated in scope, mean-spirited in approach, and epistemologically indefensible.

I begin by identifying and evaluating four points that run through Biernacki’s argument. I then demonstrate the grounds for each of my charges above. I close with some thoughts on the book as a whole and the controversy surrounding its publication.

Point 1: Textual wholism. This claim runs throughout the book, and is the most consistent ontological claim. Here’s one example, from page 11:

 [Using] ritual productions…. you can rip everyday implements or emblems out of their original contexts and find new ways for them to work as “symbols” in the cleansed space of the performance. Yesterday’s wine turns to Christ’s blood at the Last Supper or Halloween cloaks into Shakespearean robes. The coding of texts works much the same way, because the operation assumes one can convert regenerating meanings into an isolated token, a datum label. It dissolves an opus’s multilayered architecture and resonances into a discrete, single-level cipher (or linear string of ciphers) that can be brought into previously unimaginable relations to each other.

By page 16, it has become a programme:

that we not grant epistemological legitimacy to the interpretive model with which we start; that there is no pigeon-holing of textual evidence…; meaning is bound to concrete, unsystematizable prototypes; and, finally, that we read the texts as they are embedded in a culture different from the one by which we create our ideal models as investigators….[in order to] guard against over-rationalizing methods that confuse investigators’ analytic categories with those naturally “in” the evidence.

And later: “Dramatic omission and suggestive possibility in narratives stimulate our engagement as readers but render consistent coding nearly impossible” (37).

There are two pieces to textual wholism as presented here. The first is internal: the meanings of words and phrases are wholly dependent upon the remainder of the text in which they are set, so analyzing them outside that context evacuates them of meaning; “fancifully expung[ing] from view the multiple and idiosyncratic communicative projects inside of which each text feature was originally configured to function” (24). The second is external: texts themselves are set within historical and cultural contexts, so they must be read “as they are embedded in a culture different from” the analyst’s.

Both of these claims are absurd. The former is absurd because it assumes an untenable theory of language. In order for the claim to be correct, it must be the case that words and phrases have no meaning outside their textual context. But that is simply false; while the specific meanings are conditioned, of course, on context and sequencing, words and phrases also reference meanings from outside a given text. Indeed, that’s probably the most important source of meaning in written language! The latter claim is absurd for reasons that ought to be obvious: the claim that texts can only be interpreted within their “own” cultures would leave all texts effectively without meaning, since one of the core properties of texts is that they are portable across space and time. Any interpretive act involves reinterpreting a text in a new context: that of the reader. Hence it is fatuous to claim that there are some categories “naturally ‘in’ the evidence” and others imposed from without. Indeed, Biernacki ought to know as much, since the genius of The Fabrication of Labor emerges precisely from the practice of reading texts from outside their “own” cultures to bring them into comparative relief.

Point 2: Scholars ought to document and publish their coding assignments and decisions.

“Everyone,” writes Biernacki with his characteristic overreach, “(researcher, reader, dean) insulates what transpires in coding as an unapproachable black box. Investigators vouch for their ‘data’ but do not necessarily release information about the sample sources, let alone the goings-on by which they translated texts into outputted variables” (19). With respect to Bearman and Stovel, he writes: “The ‘data’ are interpretive guesses all the way down, with no bottoming out in ascertainable givens” (36).

This point is technical and pragmatic, not epistemological. Ironically, it is a point far more comfortable in the realm of social science (the latter word always enclosed in scare quotes here, as if the very mention of science is taboo) than in Biernacki’s preferred hermeneutic approach. Yes, scholars should make their data available and their procedures transparent. It appears from Biernacki’s account that these three scholars did not do so particularly well, though it bears mentioning that between 11 and 25 years passed between the works’ publication and Biernacki’s replication.

Professors Bearman and Griswold undertook to locate their coding records, and they were patient in communicating about their efforts…. Transparency of data in a research community is championed by journals and by granting agencies. In the cross-genre of qualitative coding, this norm has become largely inoperable.

So says he – with N=3, and after considerable time had passed. (Bearman wrote me that “The box with the stories really did become a home for squirrels.”)

I doubt if anyone would disagree that transparency would be better. But there’s nothing inherent to coding that makes transparency impossible. In fact, computer technology and the price of storage have improved enough to make it easier since the studies Biernacki critiques. Biernacki inflates what are at worst sloppy practices into methodological crimes.

Point 3: Studies using coding are presented as authoritative.

“’Becoming a Nazi’ stages the discovery of network theory as the model of intelligibility for all facets of social experience,” writes Biernacki (56), with a footnote that is nearly totally irrelevant to the statement (it refers to p. 51 of Algirdas Julien Greimas’s The Social Sciences: A Semiotic View. I can find nothing on that page that is relevant to Biernacki’s claim, but I’ve posted the page here for anybody who wants to look.)

But, in fact, the article does no such thing. Rather (from the abstract):

By representing complex event sequences as networks, inducing ‘narrative networks’, it is possible to observe and measure new structural features of narratives…. The substantive idea that we develop in this article is that the observable narrative structure of life stories can provide insight into the process of identity formation.

Or, as Bearman wrote me directly, “The article is very clear about what it is — a picaresque journey.”

Biernacki:

I have mocked the pretensions of a cross-dressing social ‘science,’ not those of conventional natual science, whose clothing sociologists try to wear as their own by presenting ‘large-N’ coding results…. Certainly it [the inevitable handpicking of sundry sources] is more honest than reifying a mechanical en masse choice as natural. (154-155)

This is a straw man, as I’ve demonstrated in the case of Bearman and Stovel above. None of the three works in question actually presents itself as definitive or scientifically totalizing. On this point, Biernacki’s breathless accusation and huffy superiority strongly suggests an inferiority complex on the part of the critic.

Point 4: Each of the three studies made problematic interpretive choices.

Within each replication, there are numerous examples of problems Biernacki locates with the sampling or analytic decisions. One example among many: “For all we know, had Bearman and Stovel calculated the role of anti-Semitism in comparison to variables defined more concretely, anti-Semitism might pop out as ‘high’ in power centrality” (44). There is an extended discussion of selectivity in quote selection by Griswold, who “expounded on how several reviews treated the Trumper ‘scene’ in a novel, whereas most of her preferred examples barely alluded to it” (128).

Biernacki’s criticisms here seem generally believable, but given the overall hostile tone of the book and the fact that there is no response from the authors, I’d tend to withhold judgment on the specifics. Evans has provided a comprehensive, and IMHO convincing, response in the prior book.

The important point here, though, is that Biernacki’s critique of the analytic decisions presupposes that there is an appropriate and valid way of coding texts for analysis. If there weren’t, there would be no adequate grounds for launching the critiques! To the extent that this point is correct, point 1 cannot be, and vice versa.

–*–

Now let me turn to each of the problems I see with Renventing Evidence.

1. It is pompous in style. Particularly in his extended, disingenuous re-casting of coding-based research as “ritual,” it is consistently dismissive and holier-than-thou. “Only in liturgy,” sniffs Biernacki as one of countless examples, “is the truth of expression decided in such a fashion, by whether the outside world fits the ritual, not by whether the ritual fits the outside world” (69). Or: “Brought to life in social ‘science’ as ritual, familiar instruments of explanation act as if possessed by foreign spirits” (99). “The first step in critical diagnosing of a coding diagram may be insisting on sheer absence of sense” (125). Dan Hirschman is charitable in describing the prose as “dense,” which it is not particularly. Actually it’s quite readable. What it is is pompous.
2. Its evidence is muddled. Through meticulous attempts at replication, Biernacki describes flaws in the execution of three coding-based studies. Based on these critiques—which go unanswered by the studies’ authors, so we have no adequate way of adjudicating them—he seeks to establish an epistemological dismissal of coding as a theoretical practice. But not only is this dismissal unwarranted by the evidence, the very structure of the critiques themselves actually demonstrates why the dismissal is incorrect.
3. It is vastly overstated in scope. The title of the book is “Reinventing Evidence in Social Inquiry.” One might think, then, that the book would have something—anything—to say about evidence in some other area of social inquiry. But not so. According to JSTor Data for Research, the word “coding” appears in about 2.7% of sociology articles in 1987 (when Griswold’s article appeared), climbing to 4.7% in 2001 when Evans’s book came out, and to a high of 7.3% in 2011, the last year for which full data are available. Providing a half-baked critique of three relatively old exemplars of a style of sociology comprising well under 10% of the field is hardly reinventing evidence in social inquiry.
4. It is mean-spirited in approach. Each of the three studies reviewed is treated as utterly without merit, its author(s) apparently ignorant or worse. I assume there is some back-story explaining Biernacki’s antipathy to his colleague and department chair, John Evans, and of course the infamous dean’s warning not to publish the book is wrong on its face. But whatever its source, that antipathy has now led to two extended debates and a scorched-earth style here without any meaningful progress
5. Finally, it is epistemologically indefensible. The main epistemological claim is wholism: that decontextualized fragments of text have no meaning separate from the surrounding text, and that in turn texts have no meaning separate from the cultural milieux in which they were produced.

• On the first: “The ‘representative sample’ approach of science cannot accommodate the pursuit of exchanges of cultural significance” (62). “Word frequencies never have a legible relation to cultural competencies, repertoires, or semiotic systems. Numbers do not sketch anything of verbal significance” (143). This is a bizarre theory of language, utterly irreconcilable with any reasonable understanding of how meaning is encapsulated in and extracted from text. Taking it to its logical conclusion, it is an indictment of any kind of analysis (the root, ana-, referring to dismantling an object of interest to understand its parts). All science is about discarding some information in order better to understand the information that remains.
• On the second: the requirement that texts be interpreted in their own cultures would reduce the meaning in such texts to nothing—the whole point of written text is that it is transportable across time and space and can be interpreted elsewhere in ways that bridge author, reader, and the respective cultural environments of each. The ridiculous requirement that texts be read within their “own” cultures quickly reduces to Borges’ fable of the map.

An arrogant exercise in puffery, Reinventing Evidence offers precious little of value for understanding or reforming social inquiry of any stripe.