what does it mean to say an algorithm is racist?

Yesterday, tipped off by Beth Berman, I posted a screenshot of a pair of Google search results onto Twitter. The screenshot (below the cut) shows what happened when you searched for “professional hairstyles for work” and “unprofessional hairstyles for work”. I labeled the screenshot “This is what a racist algorithm looks like.” (BoingBoing picked up the story around the same time, and seems to have traced back the idea for it to the original source.)


The tweet got a lot of reactions. Several commenters noted that the actual search results for the second search included primarily discussions of why the pictured hairstyles were not and should not be considered unprofessional – that is, anti-racist affirmations, not racist comments.

More generally, I want to reflect on what it means to call an algorithm racist. Cathy O’Neil has some great commentary on this same case under the heading I’ll stop calling algorithms racist when you stop anthropomorphizing AI. In other words, she argues that because we talk about AI as if they were people making decisions, it becomes reasonable to call those decisions racist.

I wrote up some of my thoughts on the issue on twitter and I wanted to record them here. I’d also love to hear your thoughts about what it means to call an algorithm racist, and whether and how that discourse is useful. Here’s what I wrote (lightly edited):

I’ve had a lot of responses to the tweet of the form “the data are racist, not the algorithm.” Google uses algorithms to answer questions. When an algorithm returns a racist response, I think it’s fair to call the algorithm racist. Understanding why the algorithm returns a racist response is important, but that doesn’t change the fact of the racist response. Part of the confusion is that some equate racism with conscious prejudice. But neither intent nor awareness are required for racism.

Beyond that, I’m not actually sure the data are racist in this case. Many of the images of black women associated with “unprofessional” are from sites critical of that claim (for example). But the algorithm still associates the term “unprofessional” and the image, ignoring the critical discourse or negative modifier. So the data here are discussions of racism, but they aren’t exactly “racist data.” As is made clear from the text summaries. So I think the defense of “racist data, not racist algorithm” rings especially hollow here.

What do you think?

Author: Dan Hirschman

I am a sociologist interested in the use of numbers in organizations, markets, and policy. For more info, see here.

8 thoughts on “what does it mean to say an algorithm is racist?”

  1. So I’m of the opinion that we use the word “algorithm” way too much these days, to the point when we overlook other terms that are perhaps more constructive: software, interfaces, design, and so on. In your particular example–which is a brilliant illustration, by the way, I am totally going to use it as a pedagogical example in class–I might frame the question as an issue of design. So as you point out, those pictures are actually parts of articles that are critiques of racism, but the design format of Google Image search only shows us the images and omits the (con)text around it. That way we could turn it into a question about better design, one where we learn from our mistakes, and ask: might an image search not be better off if the text around the images was also reproduced in the search results? Maybe I’m wrong about this, but I feel that discussions around design can tend to go in more constructive directions than discussions around algorithms (mostly because algorithms are (a) black-boxes, (b) proprietary, and (c) the word itself is notoriously hard to define.)

    Liked by 2 people

    1. I couldn’t agree with you more about the overuse of the term algorithm. I’ve been trying to explain this problem to a coauthor, do you have any recommended readings about the problem of polysemy here? I’d love to see something on the order of Abend’s “Meaning of Theory” for algorithms…


  2. IT firms are now creating automated (see I did not use the word algorithm) resume and job application screening software. Under discrimination law if they use a screen which is correlated with race or gender, but not directly required for the tasks needed to do the work, they would be discriminatory. Because they are proprietary algorithms (oops) it is often unclear what is under the hood even to the employer who buys the service. Under the law if the algorithm returns a race biased result it should be illegal. Off course, many employers are already doing the same thing with credit scores and without fancy software. Also probably illegal, but not yet contested.


  3. Algorithmic (sorry) racism seems to be mostly a variation on institutional racism, no? Policies that have no explicit racial component nevertheless have racially disparate impacts, as when state-level school funding formulas are based on tax revenues and number of enrollments and so on but benefit mostly-white school districts because the population is already racially segregated. The response of “the algorithm’s not racist, it’s just reflecting the world” begs for an explanation of how rules that aren’t explicitly about race can worsen existing racial disparities — thus making them racist, even without intent. (Not that I’d attempt that with randos on Twitter.)

    The question remains of whether there’s something distinctive about algorithmic racism vs. other kinds of institutional racism. It’s opaque, but institutional racism can be pretty opaque, too. (And sometimes it’s not so opaque.) I guess at first glance I’m not convinced that it requires fundamentally different conceptual tools for thinking about it, although clearly the context is new and some of both the challenges (the black-box nature of the algorithm) and the solutions (audits) may be distinctive.


    1. This was my instinct as well. The result reminds me very much of the outcome of “color blind” approaches to racism elsewhere, as in the John Roberts notion that “The way to stop discrimination on the basis of race is to stop discriminating on the basis of race.” If you are designing any automated process within the context of a society with racism, and you DON’T account for that in your design, your product can be expected to reproduce racism in some form or another. Sarah Jeong makes much the same point about AI here


  4. Search and other AI algorithms are the most literal embodiment of statistical discrimination there is. Racial prejudice is in my view the reification of statistical propensities with the claim that the world ought to be the way an algorithm predicts. Google can’t do that. Only people can.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s