Last year, I had the honor of chairing two article award committees for ASA: the Granovetter Award for the Economic Sociology Section and the Junior Theorist Award for the Theory Section. Both committees ended up similar processes that worked fairly well, so I thought I would share a brief description in case it’s helpful to any future award committee chairs taking on the task for the first time.
The context of these awards is that individuals (mostly) self-nominate papers, and a committee of ~4 scholars reads them and pick 1-2 winners and 0-2 honorable mentions (roughly, rules vary by section). The number of submissions can range quite a bit; one committee I was on had something like 80 submissions, while other committees (including the two I chaired) had more like 25-45. Because of the large ratio of submissions to awards (e.g. potentially 80 submissions to 1 winner), the task seems to me one of “highlighting the best” rather than “accurately grading the papers”. That is, unlike (say) grading 80 student papers, it’s not really relevant whether a paper is a B+ or an A-. What matters is identifying the A+s and then deciding which of those most merit recognition. I won’t detail the precise criteria we used because they differed a fair bit across the two committees, but I do think it’s important for the committee to explicitly discuss different dimensions (novelty, rigor, empirical vs. theoretical payoff, quality of the writing, whatever else comes to mind!). With that in mind, here’s the process we used.
In the first round, each paper is assigned to two readers. These readers give the paper a “yes”, “maybe”, or “no” as to whether it will advance to the second round. Each paper that receives at least one yes or two maybes advances to round 2. I instructed each committee member to aim for about 3 “yes” and 3 “maybe” to try to get a reasonably short short list, which overall worked very well and (I think) streamlined our process tremendously as compared to a traditional ranking.
In the second round, each paper is read by the entire committee (which will be a re-read for two of the members). Then each paper is ranked from 1 to N where N is the length of the shortlist (for example, in one committee we had a 9 paper short list).
The final decisions are then made at a single real-time meeting. In advance of the meeting, the committee chair looks over the rankings and proposes an agenda (such as, drop consideration of these papers, etc.). For example, there might be a tier of clear favorites, or there might be a single favorite, etc. On one committee, for example, one paper had three number one rankings and one number three and so the debate largely ended up being what to do with the four papers that all scored clearly well but not as well as that (honorable mentions, etc.). On another committee, there was much more disagreement about the winner. Either way, everyone then meets to discuss, based on that framing of the agenda. The committee then ideally unanimously endorses the final decision.
And that’s that! But I only have an n=2 of chairing these committees, and serving on another few. What do you all think? What’s worked well for you?
scatterplot 
Oh, also, I should add a logistical tip: make a single shared dropbox/box folder for all the submissions, number them as they come in, then create spreadsheets using that numbering for the scoring. Don’t ask everyone to save the papers themselves and then have all sorts of logistical annoyances harmonizing, etc.
LikeLiked by 1 person