Yesterday, tipped off by Beth Berman, I posted a screenshot of a pair of Google search results onto Twitter. The screenshot (below the cut) shows what happened when you searched for “professional hairstyles for work” and “unprofessional hairstyles for work”. I labeled the screenshot “This is what a racist algorithm looks like.” (BoingBoing picked up the story around the same time, and seems to have traced back the idea for it to the original source.)
The tweet got a lot of reactions. Several commenters noted that the actual search results for the second search included primarily discussions of why the pictured hairstyles were not and should not be considered unprofessional – that is, anti-racist affirmations, not racist comments.
More generally, I want to reflect on what it means to call an algorithm racist. Cathy O’Neil has some great commentary on this same case under the heading I’ll stop calling algorithms racist when you stop anthropomorphizing AI. In other words, she argues that because we talk about AI as if they were people making decisions, it becomes reasonable to call those decisions racist.
I wrote up some of my thoughts on the issue on twitter and I wanted to record them here. I’d also love to hear your thoughts about what it means to call an algorithm racist, and whether and how that discourse is useful. Here’s what I wrote (lightly edited):
I’ve had a lot of responses to the tweet of the form “the data are racist, not the algorithm.” Google uses algorithms to answer questions. When an algorithm returns a racist response, I think it’s fair to call the algorithm racist. Understanding why the algorithm returns a racist response is important, but that doesn’t change the fact of the racist response. Part of the confusion is that some equate racism with conscious prejudice. But neither intent nor awareness are required for racism.
Beyond that, I’m not actually sure the data are racist in this case. Many of the images of black women associated with “unprofessional” are from sites critical of that claim (for example). But the algorithm still associates the term “unprofessional” and the image, ignoring the critical discourse or negative modifier. So the data here are discussions of racism, but they aren’t exactly “racist data.” As is made clear from the text summaries. So I think the defense of “racist data, not racist algorithm” rings especially hollow here.
What do you think?