qualitative research

So there has been a bit of a debate going on in qualitative circles about funding qualitative research projects. The NSF report “Workshop on Interdisciplinary Standards for Systematic Qualitative Research”  recently was released. A follow-up from an earlier commission, this report is meant to serve as a guide for qualitative researchers on how to get funding. It was produced by a group of top qualitative scholars. Then Howard Becker wrote a response. The response is a charged critique of the NSF commission, led by Michele Lamont*.
Becker’s argument is that the NSF guidelines get the practice of research wrong. He claims, “[the report] might be summarized as, “Quit whining and learn to do real science by stating theoretically derived, testable hypotheses, with methods of data gathering and analysis specified before entering the field. Then you’ll get NSF grants like the real scientists do.”” I think he’s largely accurate in this characterization. His response is: ethnographic research isn’t done this way. He outlines a series of classic studies and shows how none of them do research in this way. Indeed, the research is far more emergent than hypothesis testing suggestions.

I was recently in touch with Becker about his response because I was denied a small (trivial) internal grant by my own Vice President’s office. My rejection letter stated,

A serious research proposal should go beyond your impressions of and personal history with one institution. If it does not, it will remain at the level of anecdotal, single-case evidence, and will count as autobiography rather than systematic research. The Committee indicated that it would be willing to re-review the proposal if you would be willing to rewrite it addressing this concern and providing more specific hypotheses, a greater range of evidence in support of them and an evidential base that does not rely just on one institution and your personal relation to it.

In other words, my ethnography was not real research. Luckily, my department has been very quick to challenge this and back up my research as “real.” Yet oddly some of the response has hinged not on the actual practice of the research but instead the claim that legitimate gate-keeping institions have accepted my work (I have a book contract from Princeton). This encounter is the first time I’ve ever confronted the “scientific problem” with my work, and I suspect it’s in part because the review committee likely had few social scientists on it.

It strikes me that Becker is right in noting how the NSF guidelines suggest that we qualitative researchers, “act like real scientists.” Becker’s point is that qualitative research doesn’t work this way. He might drive the point further and argue that “real science” doesn’t look that way either. The implicit claim is that it is harmful to young researchers to think about their research projects in this way. Where he is (admittedly) silent is on the question of how one should write a proposal and grant money to folks given that the research practice doesn’t look like this.

As for what I’m doing, one colleague who shall remain nameless offered to give me an equivalent amount to the small internal grant if I replied:

“Dear Sir/Madam –

If I were to show you a pig that spoke English, would you demand to see a second before you felt it was a worthwhile finding?

Sincerely,

Shamus Rahman Khan”

That I won’t do. But it did make me laugh.

* Relatedly, I’ve been meaning to write a post on Michele Lamont’s, “How Professors Think“. It’s in the works. Looking inside the world of grant judgement is quite fascinating. Perhaps this weekend…

13 thoughts on “qualitative research”

  1. “Autobiography rather than serious research.” To be honest, I have read some so-called ethnographies that seemed to be more about the author’s feelings than the purported objects/subjects of the research. And some in which the problem of researcher reactivity seemed to be a really big issue. But assuming that what is in the text is description of what you observed other people doing and its implications, and you were not just recording your own actions or interactions, the idea that all single-case research involving participant observation is by definition autobiography is absurd, as you note. And, as you note, the question of how and where the data are collected is a totally different issue from deduction and hypotheses and all that.

    Like

  2. I found the point that Becker raised in the conclusion, that “NSF has an apparently inflexible rule that grants will not be given for faculty time released from teaching,” did as much to highlight the NSF’s distinction between “real science” (likely in a lab on campus) and ethnography as the bulk of Becker’s argument. He notes, “The materials for recording, storing and analyzing interviews and field notes are cheap. Qualitative researchers need money to pay for their time, so that they can make observations and conduct interviews and get those data down in a permanent form. And NSF won’t pay for that.”

    Like

  3. Is it really that “qualitative research doesn’t work this way,” or is that “not all research works that way”? Seems to me the qual/quant distinction doesn’t provide the whole distinction. Aren’t there hypothesis-testing qualitative studies?

    Like

  4. As somebody who has served as primary reviewer for ~40 NSF proposals this year: it’s pretty hard to look at two proposals, one of which the author(s) present a clear idea of what they are going to do and what might come out of it, and the other is extremely vague about both, and to feel good about recommending funding for the latter over the former. It has been surprising to me how many quant NSF proposals have quite vaguely articulated research plans.

    Like

  5. In the NSF document I find the following two recommendations (From Appendix 6: Sociology, pg 143) to be both ironic and telling:

    anticipate reviewer objections and respond to them (e.g., if you will not be generalizing beyond your data, explain why; clarify why selecting on the dependent variable is not a problem, if this is the case; explain why bias is not an issue in the study; justify why you are selecting a unique case; explain why snowball sampling is a good strategy for your study)

    Followed by this 3 bullets later:

    avoid discussing intricate matters of the philosophy of science or theory that undergird the proposal unless these are directly relevant to the research design.

    Translation: Deductive standard causal analysis facilitated through statistical comparisons of units drawn from properly drawn probability samples is the norm in science. If you wish to sample purposefully in order to describe thickly, with the intent to explicate processes or meaning structures, you’re gonna have to convince the reviewers that your work is scientific. BUT… you should not go back to the first principles of the philosophy of science to show that the assumptions supporting a concept such as “sampling on the dependent variable” are actually rooted to a fairly narrow conception of the scientific act.

    FYI, Mario Small’s short paper in the report (pg 165-171) is quite useful to illustrate the point that I tried to make here. (Useful in the sense that he, more ably than I, shows the faulty logic in assuming random selection from a known university is always preferable to purposive or theoretical case selection).

    Like

  6. oops…
    …assuming random selection from a known university…

    should be
    assuming random selection from a known universe.

    Like

  7. Has NSF ever released rates at which proposals to NSF sociology are funded, broken down by primary method used in the research?

    Like

  8. 1) Qualitative research does not equal ethnographic research.

    2) I had absolutely no idea that deductively derived testable hypotheses were so controversial.

    Like

  9. One danger that I see in Becker’s proposal is that it be interpreted as meaning that NSF needs to “trust” researchers in order to give them money. How would NSF decide who is trustworthy without an articulated story about how the social scientist will carry out the research? One option is to give grants only to people with research experience or a letter of recommendation from an advisor with experience. But it seems like it would make the funding even more biased toward a few elite or elite-connected researchers. So the question is (and I really mean it as a question, not a criticism): is there alternative ways to write a proposal that can show NSF that the work will be serious?

    Like

  10. From a recent article in the local paper about a communications professor:

    Brown has carved a niche for herself at UA teaching radio and TV writing, fantasy and science fiction in TV and film and more esoteric subjects such as qualitative research.

    Classic.

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.