I’m writing some stuff related to measurement, and specifically about operationalizing concepts as measures. Anybody have any favorite examples of clever ideas in measurement?
A economist colleague here at Northwestern (David Figlio) has a paper in which he was interested in whether African-American kids with first names like Da’Quan or Jacquizz had worse outcomes (possibly due to differential treatment by teachers) than African-American kids named Michael or Stephen. My recollection is that it was nicely designed in that the effects were identified off sibling variations: that is, families with a kid named Michael and a kid named Da’Quan. For measures, he used features of names like whether they included an apostrophe and prefixes (e.g., Lo-) or suffixes (e.g., -isha), but this didn’t take into account the apparent common feature of longer names with low-frequency consonants. So, he also included the Scrabble score of each name, which did indeed prove to be an informative measure.
I’d love it if there was a “scrabble()” function in R or Perl that assigned the Scrabble score of a string.
LikeLike
ps, another good example of creative operationalization is chris bail (who is finishing at Harvard and about to start at Mich RWJ) used plagiarism software to identify when journalists quoted or paraphrased from press releases.
LikeLike
I’m not sure this qualifies as “clever” and it certainly isn’t novel as it has been around a long time, but the Berry et al “citizen ideology” scale in political science correlates surprisingly well with a lot of stuff despite (to my mind) utterly lacking face validity in its construction. It is complicated in the details but the basic idea is using the ADA and COPE ratings of members of Congress, weighting them by the number of people in a state who voted for them, assigning the proportion of the electorate who voted for someone who was never in Congress the average score for loser’s party in the state, and calculating the weighted average for each state/year.
Their “institutional ideology” scale follows a similar procedure for state legislatures and governors, who are allocated the average ideology of their state’s Congressional representatives from the same party and then weighted by a complicated measure of political control. So far it does not seem to correlate very well with anything I’m studying.
LikeLike
Compete for the Freese prize against something that combines methods and Scrabble? I pass.
LikeLike
Figlio is great… I like a lot of his work, e.g. his clever instrument for the effect of disruptive students on student test scores… If a boy in your class has a gender-ambiguous name he’s more likely to disrupt. Admittedly, I haven’t read this carefully and its almost too good to be true, but it sure is smart.
I could probably come up with 100 different examples, but I’ll name just one more: Jesse Shapiro’s work on political slant of newspapers.
LikeLike
i’m also a fan of Gentzkow and Shapiro. however, since the issue is operationalization, it’s worth noting that the variable actually comes from Groseclose and Milyo.
LikeLike