a freakonomic riddle

Fabio recently linked to an argument by the economist historian Deirdre McCloskey against using outside letters for tenure decisions.  The argument is mostly economic: because there are no incentives for outside professors to be exacting or to say anything negative, the letters convey virtually no information because either the outside person is going to be positive or they will turn down the request.  Plus, the most reliable strategy for trying to read through the hagiography in letters of recommendation within a discipline—compare degrees of positiveness in multiple letters written by the same person—is not really available, especially when the tenure case is at the ad hoc level (the level after the person has received a positive vote from their department, where the case is reviewed by a committee of scholars from other departments).

So I was talking to a senior economist recently about the ad hoc process and asked him if his experience was that outside letters were not informative.  He said:  “Well, I’ve heard more than once that economists are more willing to be negative in their letters than other disciplines.”

A puzzle if it’s true: the economic argument is that you can’t expect someone to be negative in an outside letter because there are no incentives to be negative, and yet it’s the economists who behave least like the economic prediction.  Why?

Clue?:  My feeling when the economist was telling me this he was saying it with pride.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

19 thoughts on “a freakonomic riddle”

  1. I was on the social science divisional committee, and I can report that economics letters on the average tend to be less positive than those from any other field. It is also not unknown to get negative letters in sociology. Why is another question.

    Like

  2. How is this different than job search letters? They’re all positive. I was on a search for the first time this year. I looked for two things: (1) if someone wrote more than one letter, so I could get a sense of the types of letters they write, (2) degree. Everyone is awesome. Some are just incredibly awesome. Isn’t it the same for tenure letters? There’s very little variation. You just learn how to read the fine distinctions within that small variation.

    Like

  3. The behavior of actual economists is no more governed by this or that economic argument than those philosophers who believe the passage of time is an illusion, or those pure network structuralists who believe the concept of the person is a category error, feel compelled by the force of their own arguments to refuse to cash their monthly paychecks.

    Like

  4. Kieran: Yes, but this is a case where the economist is actually less in line with the supposed economic prediction than other people, akin to network structuralists being more likely to fight about being first author than other scholars.

    Like

  5. Shakha: McCloskey would probably make the same argument with job search letters, but I think he would recognize that it’s a bit weaker because the person has less of an objective record to go on. His argument reads strangely to sociologists, I would imagine, because of its presumption that fellow colleagues at a university all feel comfortable with each other’s competence reading work in whatever area.

    More generally, I’ll admit that I think the argument that the fine distinctions in job market letters can be read with high accuracy is largely illusory. The variation among writers is so large. So then you really are left with comparing multiple letters by the same person–and, to my knowledge, departments generally don’t save letters from year-to-year–and comparing letters across authors is a very murky game.

    Like

  6. My worry about a departmental evaluation is that it strikes me as more likely that it will be on the basis of whether or not they like you rather than the work.

    Donald is now Dierdre. So the pronoun is “she”! That’s more snarky than I want it to be. But I tried to re-write the sentence and failed. So my qualifier will have to do…

    Like

  7. Actually, her name is Deirdre not Dierdre.

    Jeremy, I agree, much depends on the letter writer. People have very different styles and it’s hard to know how any particular letter writer would describe the most brilliant student ever. This is why I think one has to be cautious about trying to read too much into what is and is not in those letters. Fortunately, there is usually more than one so that helps a bit.

    Like

  8. What is McCloskey’s rationale for why anyone would ever agree to write such a letter to begin with? Wouldn’t it make sense for everyone to always say no when asked to write a tenure evaluation letter to save themselves some time?
    My view is tenure evaluation letters do tend to have more information (and are more often openly negative) than job market recommendation letters. Tenure letters are much closer to confidential and are seen by a much smaller pool of people than recommendation letters. And job market recommendation letters are solicited by the person who is evaluated, and it is a bit to the benefit of the evaluators that the person they are recommending (*their* student) do well on the market. Tenure letters are solicited by the department or some kind of university committee, and usually try to include several evaluators with no personal relationship with person who is evaluated.

    Like

  9. The refusal to write a letter is inevitably coded as a negative evaluation in a tenure file, and thus divisional/college review committees keep tally of refusals. This ups the ante when you say no.

    The only real information you get from a letter that you could not get from reading the person’s cv and work is whether the writer is willing to invest some of his/her reputation in backing the candidate. This is why declining to write is coded as a no. Letters are only useful within a network where you care about your reputation as an honest letter writer, and the status, prestige, skill and temperament of the letter-writer are big factors in their quality and value. (As one senior colleague explained it, letters are useful because “you cannot lie to your friends.”) That is, it is a highly personalistic mode of gaining information about someone, and I really hate it when I see colleagues picking over the letters with a fine tooth comb, looking for clues to the letter-writer’s less than full endorsement of the candidate. I hate it a lot in the hiring process, and I hate it even more in the tenure review process. For hiring new PhDs, the work is often not there yet, and that is why trusted senior people’s endorsement of potential can carry weight. But for a tenure case, I see no justification, and I always thought it was wrong of academics to roll over on this in the 1970s when external review was first instituted by deans who were unwilling to trust their faculty. Is there any evidence that this has led to any global improvement in the quality of scholarship since the 1970s? The only real rebuttal to my argument I’ve heard is from people who do not trust the people in departments to make their own judgments about the work, either because they think personal networks will lead them to promote everyone, or personal biases will lead to biased evaluations. But my answer to this is that all we’ve done is to externalize the distrust and increase the amount of work all of us have to do reading tenure dossiers not only for our own departments, but for everybody else’s.

    These letters are the lineal descendants of “letters of introduction” which were used when communication was slow, to substitute for the knowledge of one’s character that would be available in the face-to-face networks of one’s home city.

    * Some people try to get around this by having a standard reply: I have a policy of agreeing to do the first X of these that I am asked to do a year, and of refusing to do all subsequent requests. But even that makes it clear that the target is not of sufficient value to you that you are willing to make an exception.

    Like

  10. Should have added that letters also get evaluated by how many pages you were willing to write, another measure of how much you were willing to invest in the candidate.

    Like

  11. OW: I know people say that about the rate of letter refusals, but do you think a committee could really use it as the basis for decision-making? (I’m not arguing, I’m genuinely asking.) I mean, it’s inference to a single case, and given all the other reasons people can decline to write letters, it would be hard to justify a such a morally freighted and potentially legally actionable decision on the grounds that the person had too many people decline to write letters.

    Like

  12. Additional data: There is great variation in how tenure letters are solicited. Some departments will let the candidate compile a list from which people are selected. The departments also ask candidates to strike people from the jury if there is a conflict of interest. In contrast, other schools take things out of the hands of candidates to reduce selection bias. Much of this happens behind the scenes and can be tough to spot.

    Size of the tenure letter pool matters. Leading programs require anywhere from 6-10 letters, while lower ranked schools often need only 3-4 letters. The latter situation is probably where you can cherry pick the best letters. It’s hard to find 10 people who are your best friend, so that’s where you get the most variation.

    Like

  13. I’ve never seen a case declined purely for this reason, but I have seen people point to a list of multiple declinations as evidence that the department had a hard time finding people willing to support the person and using that plus the relatively low status of those who did write as a basis for dissing the work. The thing is, that you do, in fact, get more ready and cheerful acceptances for people who are “stars.” But you already knew they were stars before you asked for the letters — the glowing letters don’t change the evaluation. And for the people who are not stars, well you already knew that too. But you should not have to be a star to get tenure at one of the hundreds of (thousands?) of schools in this country. If only the best 15 people in a field deserve tenure, we’d better shut down the entire academy. Tough decisions do have to be made, but I don’t see why departments (or divisions for that matter) are any less qualified to make them than “external” reviewers.

    Like

  14. I have a question that relates to a post about the use of letters of recommendation for new hires (particularly junior hires). Perhaps this would be better saved for jeremy’s weekly academic de-mythologizing post — but I have heard that part of what letter of recommendation are good for is for advisers to indicate what original work the student contributed. This doesn’t seem like it is always necessary (for instance, a dissertation based on ethnographic data collection by that person alone) but is important for the kinds of students who tend to work on larger projects where there tend to be lots of authors. I know that much of this is supposed to come across in the job talk and your meetings with people, but it would also seem important for the person writing the recommendation to delineate specifically the potential that their student has for independent research. Also it seems that there is an incentive to be truthful about that – a faculty member would not want their contribution attributed to their student at the same time that they don’t want to destroy their student’s chances of getting a job.

    I know that this isn’t immediately relevant to this particular discussion, but if there is a bank of ideas for a weekly session, maybe this is appropriate fodder for a week’s discussion…

    Like

  15. Mike, yes, I think this can be an important part of a letter. If a student played a major role in even just one part of the paper then I will certainly mention that. Similarly, if I ended up doing most of the work then I likely will skip discussing this issue in the latter, which itself can be a signal. (The tricky part is that the readers of the letter don’t know if I just don’t tend to talk about co-authorships in my letters or if I’m ommitting it in that particular case for a reason.)

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.