Update 12/7/19 I added a few mechanical suggestions at the bottom of the post
How do you decide what literature to cite when you write an article? Sociologists routinely complain that economists fail to cite the sociological work on the topics they write about. But sociologists themselves are typically citing too narrowly. Most view citations as a matter of nodding (or genuflecting) toward the “key” people or concepts with special attention to making sure to cite anybody who they think might end up reviewing the paper. In the process they focus on a few famous people (mostly White men), often those assigned in the courses taken in grad school, and/or articles published in the American Sociological Review or the American Journal of Sociology. We may also include people we know personally or have met at conferences.
These practices both tend to downplay the contributions of women and people of color and tend to exacerbate the prestige hierarchy in sociology. The #CiteBlackWomen campaign has been importantly pushing back against these practices, but even intentionally citing Black women runs into the problem that there are no race designations in citations so you have to know who is a Black woman to cite them, leading to citations of a few famous people, not a broader base of citations. Reviewers are notorious for “cite me” recommendations or the occasional suggestion of lines of research the writer has missed, but apart from that, a typical journal reviewer is not generally going to have an encyclopedic knowledge of all research in an area and it is not reasonable to expect reviewers to do their own literature searches as part of a review.
I’ve talked to students about their citation practices, and it appears that the most common approach involves sorting Google Scholar search results by number of times cited, and citing the most commonly-cited sources. This, of course, just exacerbates inequality.
So how can we get out of this box? I have been as bad as everyone else for most of my career, but I have recently been working hard to push back against the elitist bias, and want to share some of the strategies I have used to try to expand my citation networks. I urge others to share yours.
Since this post has gotten long, let me begin with my three main take-away points. The rest is strategies for implementation.
- Look for and cite the most recent work on your empirical topic, paying special attention to work coming from people who are not already highly-cited, less-prestigious institutions, and less-prestigious journals. This requires a broad search strategy, about which more below.
- If your work takes off from some key theoretical or empirical papers, look for the recent work that cites those papers in Google Scholar or Web of Science, again paying special attention to works by those who are not already elites.
- Pay attention to work in progress including dissertations, conference papers, and publications in journals your library does not subscribe to. PDFs of many of these are available online through working paper repositories (SocArXiv, ResearchGate, Academic, SSRN, and many campus or personal web sites); good libraries subscribe to the dissertation database. You can also do an Internet search for the author and email them to ask for a copy of the work and to ask whether there is a published version or related papers. Most people are happy to cooperate when you make these requests.
How do you implement these core principles? Begin by re-thinking the genuine intellectual purposes of citations, as opposed to the genuflection model. If I am doing an empirical project, I ought to know what other work has been done or is being done about that empirical project. Examples: Black activism in the 1990s and 2000s, the effects of incarceration rates on community health outcomes, animal rights or disability activism, divorce rates. Regardless of my theoretical orientation, I ought to know the basic empirical work about the “thing” I’m studying. If I’m going to do a secondary analysis of a public dataset, I ought to know what analyses have already been done using my variables of interest with that dataset.
For a theoretical project, or the theoretical part of an empirical project, I ought to know the recent work using those theoretical ideas: demographic transition, framing in social movements, political repression. I need to know how these theories have been used and discussed recently, not just how they were discussed in the 1970s or 1980s by the highly-cited writers featured on course syllabi.
Let’s take each of these separately.
For empirical work, I especially need to know what the latest work is, so I want the most recent work. This implies that I want to do multi-disciplinary searches for that empirical thing in several different citation databases to be sure I’m finding everything. This, in turn, implies that I have some system for collecting, reviewing, sorting, and storing these citations. (About which, see below.) I have met some students and young scholars who have been trained to do this routinely before launching a new project, and others who seem to be confused about why you would do this, so I will just say that if this seems obvious to you, please know that it is not obvious to everyone.
What are some problems that can come up about this? Basically, the problems are too much or too little. Some things you want to research have already been researched a lot, or the keywords that describe them are extremely common, so that searches turn up way more things than you can possibly read. Other keyword searches seem to turn up nothing at all. If you are blessed with being at an institution with a good librarian, you can get help with this, but I’ll give some tips about how I have proceeded.
I’ve mostly been working in areas where the problem is “too much”: broad keyword terms that bring up a lot of material that isn’t really relevant to what I’m looking for. This is in many ways the worst situation because you are basically sifting through a haystack looking for a few needles. My strategy here is to use a database that shows abstracts, set the display for as many abstracts as possible on a page, and scan them, looking for things that seem potentially helpful. Using my library’s “folder” option coupled with my bibliographic software, I save the citations/abstracts to things that might be relevant. When there is a hot link to the PDF of the source available and it seems likely to be useful, I download the file to my PDF folder, renaming the file author_date_journal_title so it will be easy to find it again from the citation. I download lots of citations with abstracts into my bibliographic software.
If the keywords are specific but your problem is that there has already been a lot of research on your topic, you proceed similarly, building a bibliographic database of all the work that has been done.
If the potentially relevant empirical literature is hundreds or even thousands of articles, you really cannot read them all. But you can skim abstracts to get a picture in your mind of what kinds of research have been done and how the literature seems to sort into groups or patterns. Then you think about what your own research is or could be, and develop a plan for which parts of the literature you should read more closely. As I scan abstracts, for the ones that seem possibly relevant, I also scan the PDF if I have it (or go looking for it if I don’t) as a way of figuring out what it is relevant to read more closely. Because I download abstracts and don’t want to plagiarize, my default assumption is that the abstract was written by somebody else, and I write “my notes” before adding my own notes into the abstract. After I’ve skimmed a PDF my notes will briefly say either why it does not seem relevant or additional details not evident from the abstract about method or findings. I definitely do NOT take detailed notes on every article: I focus on what I’ll need for a lit review, which is method, sample, key findings. I might use a few words to say what their theoretical focus is. If I end up writing a careful methodological review of prior empirical work, I’ll come back to the useful articles for more details.
The opposite problem is that you cannot find enough research about your specific thing, and you are not sure whether that is because nobody has done it or because you are not using the right search terms or databases to find it. Let’s say your topic is Satanic discourse among the Amish (I just made that up and actually have no idea whether Satan is a theme in Amish religious discourse) and nothing comes up in a keyword search of Satan AND Amish. Not surprising. Here you broaden your search, looking for studies of discourse about Satan in religion more broadly (a topic I’m pretty sure there is a literature on, although I’ve not done the search) and studies of the religious worldviews of the Amish and other separatist religious.
A combination of the too much and too little problem arises when the keywords for what you are looking for are broad but they just turn up irrelevancies. In this case, trying to find even one source that has what you are looking for and then looking up its citations and looking up the people who have cited that source or its citations may be your only hope. For example, I have had to dig hard to find research on the Black movement in the 1990s and 2000s because there are no useful keywords for that era and it isn’t a subject heading. But once I found a few sources (all historians!) I was able to start citation chains to find others.
For many projects, narrowing down the theoretical literature search is much harder than the empirical search. There are many theoretical traditions that have literally thousands of articles and hundreds of books and you cannot possibly read them all. But you still want to engage recent scholarship in a way that includes non-elite scholars. There are several paths into this, all of them involving a concerted attempt to examine and cite the most recent scholars.
If your primary focus is your empirical work, you can review the theoretical approaches that have been used for your empirical topic, paying special attention to the most recent and those from non-elite sources. Your theoretical literature review might overlap with your empirical literature review.
If your starting point is an older theoretical work, do a citation search to see who else has cited that work. If that brings up too many hits (it often will), narrow the search to your broad substantive area, or by adding key words to hone in on the particular angle you want to take. As with a “too broad” empirical search, you can download abstracts and scan articles to find the ones that seem more relevant, as many of the citing sources will just be “drive by” citations that do not really engage the ideas. You only need to seriously read the articles that are genuinely engaging the theoretical idea you want to engage.
As a self-serving example, I began a search for recent work on theories of repression by looking up who had cited my own work on the topic. This was a surprisingly fruitful approach, as it led me to people I had not thought of or know about who were interested in the same basic issues as I am. It was also reasonably efficient because my citations were in the low hundreds. It led me to read in new areas and to learn new things and led to my discovery of young scholars, many of them scholars of color, I might not have learned about at all, or until they had achieved enough success to pop up in the top journals. Of course, I did not stop with just the people who cited me, I then followed their citations to other sources. Thus I recommend the strategy of starting with one article or book that seems really central to your interest and finding out who cited it, as well as whom they cited.
I have talked to many young scholars who want to locate their work in “framing theory” or “political process” or “theories of repression” or “race and the state” or other broad traditions that have been going on for decades and have many complex branches. Attempts to do this nearly always end with reviewers complaining that people are over-simplifying research traditions. In these cases, you have to start with fairly recent overviews from Annual Reviews or the literature sections of recent papers or books to get oriented so that you can figure out what the most recent sub-debates are and how you might engage them. And you need to learn the vocabulary or key citations for the sub-branches of theories or conjoin those theories with search terms for your substantive area, so that you can get a manageable body of literature to review. And, again, as you do that, make a point of looking at the most recent citations and the citations to less-prestigious or specialty journals.
Learning how to do good literature reviews fast is important and not a skill that everyone has been taught. Here are some pointers.
- You need a bibliographic database. I use EndNote because I started with it and have invested in it and can afford it. It works well for me. Many other people use Zotero and swear by it. There is also Mendeley, which I’ve heard less praise for. If you are not using a bibliographic data base, you need to set aside half a day to a day to research the ones available to you, experiment with the free versions of them, and pick one. If you are at a university there should be librarians to help you with this. Or ask your friends.
- Abstracts are gold mines. I don’t like to do my primary searching in Google Scholar because it does not download abstracts. I do my primary searching in SocIndex because it covers sociology broadly and includes books, book chapters, and conference papers as well as journal articles. My work flow involves searching in databases that include abstracts and downloading those abstracts into EndNote. I then add my own notes to those abstracts. It saves a TON of typing.
- Keep PDFs in their own library. I got this tip a decade ago when I complained on social media about PDFs all over my computer in multiple folders where I couldn’t find anything. The advice is to keep every all PDFs in one folder, rename each Author_date_Journal_Title. Then you can always find it from the citation. If you also keep your PDFs in a project-specific folder, put a back up in the PDF folder. Trust me. You won’t remember 10 years from now which project you were working on when you downloaded that article.
- Experiment with a workflow that works at your institution. I’m at a major university with tremendous resources. My workflow involves searching in library databases, marking files for download, and downloading citations with abstracts straight into EndNote for further review. When there are hot links to the PDFs, I download any that seem likely to be relevant and are easily accessible at the same time as I’ve downloaded a batch of citations, although the citation download often includes a hot link back to the database and, hence, to the PDF.
- I use Google Scholar a lot, especially for broad interdisciplinary searches and for “who cited this paper” searches, but it does not download abstracts, so my own workflow often involves getting a downloadable citation from the library for something I’ve found in Google scholar. I keep a lot of windows open. What worked really well for a Google scholar search was to save the search as an HTML file on my computer (but that only does one page at a time) or paste the results into a Word document, because it preserves the links, and then I can work through them one at a time, taking notes in the search and using my library sources (in another window) to download well-formatted citations with abstracts and the PDFs. Also, my library provides direct links from Google Scholar to library resources if I access Google Scholar after logging on to the library site. Yours may as well.
- Many dissertations are available on line through library subscriptions to Proquest Dissertations or other repositories. When they are not, you can email the author to ask about the dissertation or subsequent publications. You can search for the author’s name in bibliographic databases. As dissertators have generally moved on, I have had good success with searching “author name sociology” (or other keyword if they are not a sociologist).
- Read enough articles to recognize the different levels of citation, and include less-elite scholars in all of them. In this neoliberal world, citations count, even the drive-by type. The amount of time you spend taking notes on and summarizing a source depends on how you are using it.
- Drive-by citations. These are of the form “There has been a lot of research on X (e.g. citation list).” A case could be made that we ought to just eliminate these, but we include them to say, “yes, we know there is a broad body of research on this, but I can’t review it all.” I, for example, set up a paper by saying that there is a huge literature on the social construction of race that I could not do justice to in that article, but despite the detailed debates within the literature could pull out the general point that there is broad agreement that race is constructed in interaction with the construction of states and nations. And then included a long drive-by list of citations. When I look up the citations to my own work, the vast majority are drive-by, and many of those are wrong (they have either cited the wrong article for something I said/did in another article or never actually said anywhere) or entirely gratuitous (e.g. a list of random people who have used rational action frameworks). My view is that you should at least have read an abstract and preferably have skimmed before including something even in a drive-by list. NOTE: Many of these drive-by citations will be to serious foundational theoretical works that you have studied as part of gaining background in an area, but for the purposes of the article, are just assumed background and not the main point.
- Shorter 1-2 sentence citations, where you provide a brief synopsis of what the work actually found but don’t really engage it in any depth. These have place in summarizing the state of empirical work. People who have chased down citations will tell you that there are clearly a lot of people citing people who have cited people without even looking at the original, and i urge you to go back to the original and at least skim it before citing it, especially if you are saying something critical about it.
- More serious engagement, where you give more attention to methods and detailed results, or to parsing out exactly what the theoretical arguments are. Here I would hope it goes without saying that you should actually read something before critiquing it, but you’d be surprised how many people begin a literature review by critiquing the received version of some old source without actually reading it for themselves.
12/7/19 From some Twitter comments, additional thoughts/suggestions.
(1) Librarians can really help you do a lot. Don’t be shy about asking. Really. Librarians have picked up my post and sharing it among themselves and commenting/discussing about how to help people with their searches.
(2) Boolean searches can help limit your results to a higher percentage of “good” hits. If you don’t know what these are, ask a librarian. Or check out the help files in your library’s web site. I was describing a very broad search strategy because I was trying to find research that lacked good search strings, even with Boolean operators.
(3) Zotero users will probably have a different work flow than I described using EndNote. As I said in the post, although I use EndNote and am quite happy with it, I know a lot of people who really swear by Zotero. Similarly, some people prefer to link to PDFs available on the web without saving them to a local computer. Your choice, which depends on your work situation. I prefer local copies because the search time and lag time over even a high speed connection is a lot longer than local offline load times.
(4) It is not cheating to start with the literature review from other people’s publications or shared bibliographies, but you want to start there, not end there. Use it as a source of articles that you will actually search for and at least scan yourself and use as a basis for further searching. I have often found that a citing source has misconstrued the cited source and/or that there was better and more relevant information in the cited source than you’d know form the citing source.
(5) Incremental updates to a search are much less work than the initial scanning for new material. You can also set alerts in Google or other systems to be automatically told about new relevant publications.
(6) Many on-line copies of published articles include their list of references, often with hot links to the source.