selection

Among NBA players, there’s very little correlation between a player’s height and how many points they score. Michael Jordan was slightly below average height for an NBA player; Allen Iverson was in the bottom decile. And yet no one would argue the height is irrelevant for someone’s prospects at playing pro basketball: the average NBA player is nearly 6’7″, that is, in the top percentile of the male population.

The relationship between GRE and graduate outcomes is like this. If you take a group that has already been selected on a number of characteristics, including test scores, then there is often only a modest correlation between test scores and outcomes. But exactly the same could be said for any of the other criteria that were used, like GPA, or like the quality of the writing sample or the ever-amorphous assessment of “fit”, if either of the latter were to be assigned scores.

One can not extrapolate from this reasoning to draw conclusions about how people should be selected in the first place. If you did, you’d conclude that nothing departments use to admit students matters much for whether or not they will succeed, and that perhaps the fairest thing to do would just be to admit students completely at random. Admittedly, this would save departments a lot of time.

I freely confess the foregoing are issues I’m touchy about. If it weren’t for standardized tests and grades, I don’t know how I would have been able to attend college in the first place, and honestly I feel the most plausible counterfactual is me managing a convenience store back in Northwest Iowa. If it weren’t for standardized tests and grades, I’m also not sure I would have had the opportunity to pursue a PhD, or at least not at a place as good as the place I did. I do not believe my being in academia is any sort of mistake, that I’m just a lucky impostor. I have been fortunate in many, many sorts of ways, of course. But, especially at the time, how I did on the tasks available to me felt like what was actually within my own control, what I could actually work at and compete on if I wanted to distinguish myself.

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

22 thoughts on “selection”

  1. I agree with this. I often feel like trying to choose among competitive grad candidates is difficult to the point of being somewhat random BUT there’s a pretty clear line between those who are competitive and those who are not. Certainly GRE and GPA may not always be the best indicators, but they nevertheless have their uses.

    Liked by 1 person

  2. I’d like to see a data collection from soc departments on students with low GREs who were admitted. I know some succeed but I think the odds are lower. I’m wondering about the determining factors within that group.

    Like

  3. Well, there is a difference here–we can indeed tell how well non-NBA players play basketball. For example, we can looking at the college or high school game performance of those drafted vs. those not drafted, a measure basically similar as the measure of game performance for those playing in the NBA. We have no such comparison measure for non-admitted Ph.D. students–therefore, we cannot determine if the students we are admitting are indeed more likely to succeed in graduate school.

    While I do not necessarily object to the use of standardized tests in some form of graduate admissions, such as the MBA, the ability to choose the correct multiple-choice answer to vocabulary and mathematical computation questions does not seem remotely relevant to students’ ability to succeed in a Ph.D. program. A student who is excellent at standardized tests (and, for that matter, who has a high GPA) may have excellent cognitive skills in a variety of capacities, but does not necessarily have the creativity, intellectual curiosity, grit, stamina, etc. to craft a dissertation–nor does he or she even necessarily have any sense of what he or she is getting into.

    I’ve never had the opportunity to select graduate students for admission, mind you, but I am pretty sure that my student with average GRE scores and a 3.7 GPA who completed an award-winning independent research project would make a better Ph.D. candidate than someone with a 3.9 and a top 10% GRE score who has never written a paper longer than 7 pages or participated in any way in original research. That’s not to say, Jeremy, that we shouldn’t have spaces for people who came from places where original research is not a possibility (on the other hand, it is a possibility even at my underfunded undergraduate-only less selective comprehensive public college)–but it is to say that the very valid observation you make in this post is not sufficient to justify the continuation of an admissions practice that has very little direct applicability to the Ph.D. itself and which is furthermore biased against people of color and other groups who remain underrepresented in the academy.

    Liked by 1 person

    1. other groups who remain underrepresented in the academy.

      What groups? High-SES whites? Children of academics? Because those are also big beneficiaries of ignoring scores and grades in favor of looking at one paper somebody–to some unknown extent–“wrote”, where they went to college, their extracurriculars, and a personal statement. Also, regarding “people of color,” we should be honest that one thing discounting test scores and grades in college admissions does is abet considerable outright discrimination against Asians and Asian-Americans.

      Like

      1. I certainly would not argue for discounting grades–and I am not even suggesting that GREs ought to be optional. Unlike in undergraduate admissions at large universities, Ph.D. admissions committees have the luxury of considering complete application packets in their entirety. This means that low GRE scores could be considered in the context of an entire application–for a student without other supplementary evidence beyond GPA, GRE scores could provide a boost suggesting the applicant’s underlying cognitive skills, but an applicant with lower GRE scores from a background unlikely to provide test-prep resources but who has participated in a variety of activities and projects that testify to their research experiences and skills could have those scores discounted. Incidentally, it is not so much the abandonment of standardized test scores that harms Asian American applicants as it is requirements for “well-roundedness,” which I hope are not active at the Ph.D. application level, as playing in the campus orchestra and participating in a sorority are unlikely to be predictive of research success.

        In my opinion, one very useful adjustment to the entire schema would be if undergraduate transcripts came with some information about grade distributions. I understand that there are some limitations to plans in which average grades are provided for every single course, but there are fewer objections to providing the average GPA for the major or at least institution as a whole, accompanied perhaps by an indication of what percentage of students earn graduation honors and/or complete disciplinary honors projects. This would help excellent students at public comprehensive institutions, where average GPAs are much lower and with which Ph.D. admissions committees may be unfamiliar, and provide additional information to contextualize GPAs at elite institutions (where it is not uncommon for half of all graduates to graduate with A-level GPAs and graduation honors, making grades much less useful as a predictive measure and increasing the need to look at GRE scores as a corrective).

        Like

      2. Hi, Mikaila. I’m probably coming across like more of an extremist on all this than I am (but, hey, I said it was a matter about which I’m touchy). I hope I don’t come across like I think admissions should be a GRE pageant; it’s more I feel like there’s a desire to dismiss them entirely. I’m not on board with that at all, and I don’t think the evidence is either. But I think a transcript is certainly more important. I’m also pick a place in the middle of a writing sample and start reading (the middles of papers are what I think are most telling). I’d put even more emphasis on the writing sample if not for the vast differences in the amount of help people get with them.

        My pet peeve is overemphasis on aspects of personal statement unrelated to the articulation of academic interests, because it feels to me to embody a dismissive notion toward somebody’s actual body of work (yes, we probably disagree on GRE as part of somebody’s body of work), in favor of whatever buttons they manage to push in a couple pages. I’ve been in discussions in the past where some people have talked like they have mistaken the task at hand for casting a reality TV show. Dispiriting.

        Like

      3. Wow, people really write about things irrelevant to academics in personal statements, and get admitted? The things we people in non-graduate departments don’t know… Anyway, it seems to me that if you are going to be admitting people based on the personal statement, you ought to be able to clearly operationalize what you are looking for. The advantage of the GRE is that it is a well-operationalized and presumably reliable measure; I just think that giving it too large of a role, particularly in the absence of measures of students’ understanding of what Ph.D.-level work entails, their likely success at independent research, and something like grit may produce a pool of graduate students who perform exceptionally in coursework and exams and then either do not finish or finish with uninspiring and mediocre dissertations and research agendas. On the other hand, to whatever extent we as a discipline devalue original and creative research, and to whatever extent we are moving in the direction of placing graduates in consecutive post-doctoral appointments, perhaps the old measures of success are no longer as important ;-)

        Like

  4. Good points Jeremy. As often is the case, you’ve introduced an interesting topic. The debate over grad admissions is a perennial issue (and not surprisingly seasonal). But, a few friendly questions/concerns. I don’t mean to imply you didn’t consider these issues in your post – but hopefully these contribute to the debate.

    You realize you’re committing a sample selection bias by using yourself as an example, right? By any objective metric of success, you are several standard deviations above the mean and thus you’re sampling on the dependent variable. We’re not observing/discussing the many cases where someone had high GRE/GPA but failed in grad school. Or, the many cases of mediocre GRE/GPA that were successful. From my two years chairing grad admissions, I can think of many examples of both of those sets of cases.

    I just cannot imagine anyone would argue that GRE/GPA do not matter *at all*. Almost no one would dispute that there is some baseline level that GRE/GPA must be above. For example, I doubt anyone would say it is okay to admit a person with a 10th percentile GRE (or a GPA of 2.0) to a top grad program. So, what we’re debating about is if there is a relationship between GPA/GRE and success above given baseline. For example, can you really be very confident that GPA/GRE predicts success once we get above a GRE at the 75th percentile? I’m not saying it is definitely not true, I’m saying no one probably has the evidence to warrant being very confident. If we agree it is hard to be confident that is a strong relationship above the baseline, the reasonable debate should probably be about where to set the baseline. And maybe what sort of criteria would justify ever admitting someone below that baseline.

    Your Michael Jordan example is telling as his success has often been attributed him being one of the most competitive, gritty, hard-working, focused, and skilled players. Those are the very same sorts of traits that Mikalia suggests are relevant. Sure, Jordan had physical ability, but many many others had the same ability and didn’t succeed at all.

    Taking your basketball metaphor one step further, think of professional sports teams in drafts. Sports franchises have tremendously more information on past performance, skills, abilities. They also have much cleaner and more objective metrics of success, and much more readily available data. Yet, the failure/miss rates in drafts is notoriously high. Now one of the problems is that grad admissions committees are filled with people with quite confident views on how to predict success. But, there is rarely ever any accountability for predictions (7-10 years later) or much modesty about low information. Maybe what we need is a sabermetrics-like study where a grad program quantitatively tracks its success/failure in admission. It would need to be a big program that “observes” good applicants that do and do not get admitted. Then, it could track those people down 10 years later and see what predicts success.

    Finally, one dilemma is that we often don’t have consensus on how to define/measure success. I would vote for a few metrics: a) completion of PhD, b) publication, c) attainment of tenure track job, c) attainment of R1 job. As one moves up that list, we’re talking about relatively rare events in the large sample of new grad admits. So predicting such rare outcomes would also be thorny.

    Liked by 1 person

    1. Hi, Dave. Happy 2015 from Australia. Much could be said about various details of this. The paragraph about myself at the end was not intended as the substantive argument, which is just an obvious point about selection bias. I could have used other examples than the NBA, but it seemed obvious and at hand. If a selection process selects on a characteristic, then a study of the relationship between that selected characteristic and an outcome will be attenuated, and in highly competitive selection processes this attenuation can go all the way to zero. Maybe I’ll post some simulations!

      In any event, the point was not that admissions should be a GRE/GPA or SAT/GPA pageant. But, as for whether people would say that it doesn’t matter at all, there’s certainly a trend in college admissions to not require SAT, and it wouldn’t surprise me if there became a trend in grad admissions to make GRE optional. And I think you’ll certainly find in sociology there are some people who would dispute the idea of a “baseline” GRE, and more people who if they did agree with the idea, wouldn’t put that baseline at the 75th percentile.

      Michael Jordan is in the 99th percentile of men for height (at 6’6″) and I think was also extremely unusual in general population terms for athletic gifts that are hard to see as just being due to hard work and grit. This isn’t to say the stuff about him being extremely competitive and hard-working and such isn’t true, but the comparison group is the <0.1% he was in for non-psychological reasons, not the population as a whole.

      Like

  5. GPA and GRE are not the same indicator and are subject to different debates. Of course, people who are high on both tend to be strong performers, but the correlation between them is modest, at least among those who bother to apply to grad school. (The first step of the two-step selection process Jeremy alludes to.) This is partly because grades at most elite schools are very inflated and students at elite schools were already selected for top test-taking ability in the prior round.

    I’d say based on past experience that applicants with high GREs and mediocre grades are a very bad bet for success in grad school. Mediocre test scores and high grades are a better bet for at least being able to get things done.

    I have not looked at files now, but in the 1990s it became clear that the GREs were being scammed in China and Korea, and transcripts and letters of reference and writing samples were being faked in large numbers.

    Like

    1. I share the view that high GRE and mediocre grades are a giant red flag. Talking about high GRE vs mediocre GRE probably requires some operationalization of the sorts of percentiles one is intending by these terms.

      Like

  6. Good point on effect attenuation if a characteristic is used for selection.

    Yes on Jordan, but even if he was in the top .5% for ability including height and other athletic capacities, how many thousand are in that group? Tens of thousands?

    I don’t know where the typical sociologist would stand on whether to factor in GREs at all, or where to set the baseline. I’d be very surprised if sociologists at elite departments would throw out GREs altogether. I’d be more moderately surprised if they resisted a baseline (though I don’t know where they’d set it).

    Still, it is an interesting thought problem to think about the pool of applicants with a given baseline. Through this link: https://www.ets.org/gre/revised_general/scores/understand/ , you can get the percentile distribution for all GRE-takers (https://www.ets.org/s/gre/pdf/gre_guide_table1a.pdf) and specifically among those intending to study sociology (https://www.ets.org/s/gre/pdf/gre_guide_table4.pdf).

    For all GRE-takers, the 80th percentile for verbal is about 159 for verbal, about 161 for quantitative, and 4.5 for analytical writing. Among sociology-intenders, my best guess says that translates to about the 19th percentile for verbal, about the 5th percentile for quantitative, and about 30th percentile for analytical writing.

    Per year, it looks like roughly 1900 GRE-takers intend to study sociology. That means 361 have verbal scores, 152 have quantitative scores, and 570 have analytical writing scores in the top 20% of the total GRE distribution.
    I don’t know if that is a small or a large number of potential applicants for all sociology PhD (and MA) programs. If one is looking for high quantitative scores, the pool of sociology-intenders is probably not very deep (especially as those applicants will not be normally distributed across programs). But, if one is looking for high analytical writing scores or even high verbal scores, there are likely to be many applicants with >80th percentile GREs. If one wants 80th percentile on all three, I concede the pool is not very deep.

    It is also interesting to consider that with 1900 GRE-takers, let’s say 10% don’t end up applying to sociology grad school (no idea if that is a good estimate). But, that means there are 1710 people applying to sociology grad programs.

    I would presume any top department would feel happy if an admit one day got a job at a R1 sociology program. Let’s say there are 50 of those jobs per year (too high?). That means <3% of applicants will be a “successful” outcome. If success is defined as one of the ~15 jobs per year in top 30 PhD programs, we’re talking about a success rate of <1%.

    So, the job for grad admissions committees is to predict which of the 1710 applicants will make it to the top 1-3% of the pool in ~7 years. Setting that 80th percentile threshold as a baseline, you’re still talking about quite rare odds of success.

    Like

  7. David brings up the point of attenuation, though I can’t find who made this point in the earlier comments. Useful to distinguish here between causality versus correlation and prediction. Consider the example of the height of basketball players. Assume that in the general population the relationship between height and success in basket ball is .9, meaning that 81% of the variance in success is associated with height. Now take the top 1% of the height distribution. The variance in height in their subgroup will only be 1% of what it was in the original population or .81% giving a correlation of .09. Because of attenuation within the subgroup height is not very predictive of success.

    What if we had done a regression thinking that we were estimating a causal effect? In this case the regression slope (in expectation) would be the same in the general population and the 1% subgroup. Selecting on an independent variable has no effect on the expect value (or plim) of the slope coefficient. The casual relationship is the same in both populations. Causality and prediction are not the same thing.

    As to the usefulness of GRE’s, in 1990 when I was chair of Sociology at Northwestern we had an extended discussion about the predictive power of GRE’s. I wrote ETS to get all their research. I was amazed first at how bad it was. Second, the only thing that the GRE was predictive of was grades in the first year of graduate school. Don ‘t know what the research says now.

    I suspect that in most cases we do not need GREs to make admissions decisions. If someone has a GPA of 3.8 from the good school, we are likely to admit them. If they have already published, the same. The case where they may be important is the student who comes from a school one has never heard of, they have a 3.9 GPA, their recs say that this is the best student they have ever had, and the question is how well would they have done if they had gone to a strong undergraduate institution. Essentially, we have a ceiling effect on their GPA and recs. If I were making admission policy, I would say avoid GREs if possible, but there may well be cases where they are needed. Chris Winship

    Like

  8. Hi, Chris. Your point about selection on an independent variable is absolutely correct as stated, of course, although it may be a little misleading for the problem at hand. Maybe I’ll post some simulations!

    It’s wonderful that you pursued trying to look at actual evidence about predictive power while at Northwestern. Of course, one can now Google around and get a fair bit of info about studies involving both GPA and GRE, although much less for outcomes for disciplines as distal as actually receiving the PhD and beyond.

    But, for anyone else reading this thread: It is worth noting that what is elusive for a Google search are studies of the role of “this person seems like a really good fit” or “I thought their writing sample was interesting” or “I found this person’s personal statement appealing” or “This person studied abroad in three countries” in grad school success. This is part of what frustrates me: I’ve encountered people (and I’m not referencing anyone in this thread) who seem like they want to use what I think are dubious interpretations about evidence regarding GRE in order to advance criteria that have zero evidence base whatsoever.

    Liked by 1 person

      1. NU is sort of a tweener in this respect: we don’t decline to admit people because we think they are too good for us, but we very rarely get someone admitted to Princeton or Harvard. Any other school we win-some and lose-some in varying proportions. It creates some interesting math for trying to project how many of the students we admit will actually come. Maybe that could be its own post.

        Like

      1. I accidentally hit “Edit” for a comment by Phil when I meant to hit “Reply.” Sorry, Phil. Here it was: I mean we are more likely to lose people to higher-status schools when they have better GRE scores. (Not that URMs need better GREs to be admitted – it’s within the group that GREs predict who we can’t get)

        Like

      2. And this was what I meant to reply:

        A different possible explanation for that phenomenon would be that it’s not that higher-status schools value GRE more for URM students, but that higher-status schools evaluate URM students more consistently as their GRE scores increase. If so, the observable implication would that as GRE scores for URM increase, the number of different schools to which they are admitted increases at a faster rate than the returns to GRE increases for non-URM students. So then the competition would steepen quickly for URM students with higher scores, making it more likely you’d lose those people.

        Liked by 2 people

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s