I started teaching Cathy O’Neil’s book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy in my class last week. Despite being a mathematician by training (she goes by the moniker mathbabe online), the book makes a strong case for the importance of social science generally and sociology in particular.
O’Neil looks out at the land of big data and its various uses in algorithms and sees problems everywhere. Quantitative and statistical principles are badly abused in the service of “finding value” in systems, whether this be through firing bad teachers, targeting predatory loans, reducing the risk of employee turnover by using models that incorporate past mental health issues, or designing better ads to sniff out for-profit university matriculates. Wherever we look, she shows, we can find mathematical models used to eke out gains for their creators. Those gains destroy the lives of those affected by algorithms that they sometimes don’t even know exist.
Unlike treatises that declare algorithms universally bad or always good, O’Neil asks three questions to determine whether we should classify a model as a “weapon of math destruction”:
- Is the model opaque?
- Is it unfair? Does it damage or destroy lives?
- Can it scale?
These questions actually eliminate the math entirely. By doing so, O’Neil makes it possible to study WMDs by their characteristics not their content. One need not know anything about the internal workings of the model at all to attempt to answer these three empirical questions. More than any other contribution that O’Neil makes, defining the opacity-damage-scalability schema to identify WMDs as social facts makes the book valuable.
The classification also helped me realize that the failure of many of the WMDs she describes could be mitigated through the application of basic sociological principles. Sociology, I think, offers two defense systems against Weapons of Math Destruction.
Methods to study the disparate impact of mathematical algorithms provide the first defense. Our discipline has demonstrated the importance of having diverse development teams as a necessary (though insufficient) condition of fairness in the way that O’Neil uses it in her book. The tech media – and tech companies themselves – often focus on the lack of representation among staff at large tech firms as a problem because it stymies upward mobility in a growth sector of women and people of color. But, O’Neil’s work demonstrates that the downstream effects are just as important. Assuming that “what people want” is equivalent to “what people I know want” (her anecdote about the CEO who didn’t want to see ads for the University of Phoenix was priceless) can ruin the lives of those not included in those discussions. As Lauren Rivera’s work shows, overcoming these biases around cultural fit are very difficult to overcome, though she has some useful suggestions for how to do so.
The principles of audit methods seem like they could be applied in the development of algorithms to assess whether they end up being unfair. There are already overlaps in the data science world since audit studies are essentially post-hoc A/B tests that developers already use and could even be implemented using unit tests in the software development process. Having sociologists on data science teams, or computer scientists versed in sociology, could relatively easily come up with profiles to run against the algorithms based on nationally representative data to see if algorithms can, for example, really disentangle credit worthiness from racial composition in neighborhoods. As an extra step, sociologists could develop “ideal types” of profiles that would likely confuse algorithms based on close ethnographic study and representative samples. Again, these could be fed through unit tests on the algorithms to assess how well they disentangle signal from noise. O’Neil has set up a firm to conduct these audits; I hope that she will incorporate the insights of sociology when she does so.
If auditing constitutes the first-line sociological defense system, then the application of sociological theory to real-world policy problems related to algorithms could offer a second-line defense. The first-line defense assumes, to some degree, noble intentions. Unit-testing software and preventing disparate impact assumes that companies are willing to do such tests internally. It’s hard to imagine that the University of Phoenix or unscrupulous lenders would want to be bound by such rules. Instead, we are left to try to guess whether companies discriminate in unethical or illegal ways.
Sociologists have a unique opportunity to demand a seat at the table to be part of the discussion on how to regulate these WMD. The initial conversation will be dominated by tech insiders, data scientists and mathematicians like O’Neil. But O’Neil clearly shows the social implications of these methods, a topic that I have some confidence in saying that we know something about. If we had a seat at the table in the drafting of these regulations (as Zeynep Tufekci should) and we trained our students to understand how these forms of social interaction structured through algorithmic media, then we could be sure that sociological solutions are included among the inevitable technological or mathematical solutions that would be offered. The applications within sociology, while important, thus far are relatively limited and largely focus on only a few areas such as financialization.
Speaking of students, we also have the opportunity to train a cadre of professional sociologists who could confront these issues if we act relatively fast. Our discipline has typically devalued professional training over academic training (I have been as guilty as others, though exceptions exist). But I think that it would be incredibly valuable for sociologists to work to develop what a professional degree would look like that would help address the problems with Weapons of Math Destruction.
What an appropriate course of study would look like should be a topic of its own discussion. It seems clear that a training program would require some instruction in computer science, which might take away from traditional topics of research or reduce the amount of methods and theory that could be taught. The tradeoff would result in training a cadre of professionals that will be in positions to advise Congress and executive regulatory agencies on the social costs (and perhaps social benefits) of algorithmic interactions, we could influence the world in ways that we have – as yet – been reluctant to do.
Many of us come into this profession and this discipline with the hopes of seeing a better world come about because of our work. Developing these sociological defense mechanisms seems like a way to do it.
Update: I meant to include in the original post the new Master’s program in digital sociology offered at Virginia Commonwealth University started by, among others, Tressie McMillan Cottom , who reminded me of the program.