beckieball and the study of not-quite-elite selected groups

Beckieball is a new sport sweeping the country. Beckieball performance is the result of three things: height, skills, and desire, all of which are uncorrelated with one another. Let’s fire up Stata and show what the correlation matrix would look like for the population:

             |  perform   height   skills   desire
-------------+------------------------------------
     perform |   1.0000
      height |   0.5751   1.0000
      skills |   0.5751  -0.0072   1.0000
      desire |   0.5774  -0.0000  -0.0005   1.0000

Unfortunately, scouts cannot really assess desire well, so when they are picking players for professional beckieball leagues, their assessment is just based on equal parts height and skills. What happens?

Say Premier League Beckieball is comprised of the top 5% of beckieball prospects selected on this basis. Here’s what the correlation matrix looks like:

             |  perform   height   skills   desire
-------------+------------------------------------
     perform |   1.0000
      height |   0.1616   1.0000
      skills |   0.1561  -0.7603   1.0000
      desire |   0.8842   0.0060  -0.0125   1.0000

Height and skills both become less associated with performance, while the association of desire with performance increases. Also, notice that the correlation between height and skills is now strongly negative: taller pro beckieball players have less skills.

Nevertheless, even in the Premier League, it is the case that taller and more skilled players are better at beckieball than shorter and less skilled players, although, relatively speaking, this difference is much weaker than it is in the overall population.

Now say Minor League Beckieball consists of the 10% of players selected on height and skills who did not make it into the Premiere League. This is what their correlation matrix looks like:

             |  perform   height   skills   desire
-------------+------------------------------------
     perform |   1.0000
      height |   0.0464   1.0000
      skills |   0.0347  -0.9421   1.0000
      desire |   0.9705   0.0040  -0.0049   1.0000

In the Minor Leagues, it looks like performance is almost entirely a matter of desire, and that height and skills don’t matter at all. And height and skills even more strongly negatively associated with one another.

(I could carry the example to Semipro Beckieball, which takes the next 15% of people; the result is just a bit more extreme version of the matrix above.)

So, the idea I was talking about by analogy to the NBA, in which the association between height and performance weakens when you study it by looking at a group selected on multiple characteristics including height, is even stronger if you imagine a league that has people who are above average on these characteristics but not the people at the very top. And for these less-than-fully-elite leagues, being very strong on one of the selected characteristics necessarily implies weakness on a different selected characteristic (or else one would be in the elite league).

Also, if you have characteristics that are difficult or impossible to judge in the process of selecting players, but is subsequently available to observe in terms of performance, then it will look like those characteristics are overwhelmingly important for who succeeds and who fails.

But if a team concludes from this information that they should just stop caring about height and skills and just drafting their beckieball players at random–say, on the basis of written statements of how they came to be beckieball enthusiasts–the team might feel very righteous and data-driven, at least up until the point it starts getting crushed in beckieball leagues by teams that did select on height and skills and, as such, have much higher average performance.

(Again, if there is any connection of the foregoing to the matters for which the NBA analogy in previous posts was used, it is left as an exercise for the reader.)

Author: jeremy

I am the Ethel and John Lindgren Professor of Sociology and a Faculty Fellow in the Institute for Policy Research at Northwestern University.

16 thoughts on “beckieball and the study of not-quite-elite selected groups”

    1. Whoa! You are exactly right! I hadn’t thought of that problem is being equivalent to this problem, but it is. You basically turn the example around and talk about what you’d conclude about beckieball skills from looking at the people in a remedial beckieball class.

      Like

  1. Nice illustration. Lots of us believe that narratives are mostly ex post accounts/justifications/rationalizations, right? Why would we turn around and treat narratives as ex ante motivations in, “statements of how they came to be beckieball enthusiasts?”

    Maybe another way to frame this issue is in terms of necessity and sufficiency. Reading comprehension and deductive/symbolic logic are necessary but insufficient conditions for making academic contributions.

    Like

  2. I’m not sure why you would characterize as “random” the evaluation of people based on “written statements of how they came to be beckieball enthusiasts.” That seems like it might be a good way to assess “desire,” which I think you just said is one of several important predictors of success.

    Liked by 1 person

    1. It’s Saturday night, so what the heck. SOPs don’t measure desire. They measure whether people are in the right political tribe (like not having the last name Goldman used to signal). Unless I got awful SOP advice, I was instructed to keep it about research I had done and potential fit, not blather on about how bad I wanted to save the world and how hard I had it growing up.

      Like

    2. I think potentially useful things one can get out of a short written orienting statement include: (1) writing ability, (2) ability to construct an argument-like document, (3) understanding of what the person is interested in and how it might fit with the matter at hand, (4) a sense of seriousness of purpose, (5) some idea about motivation/interest, (6) some discussion of experiences that provide good background for the task at hand or some general idea about perseverance. (The accuracy of inferences that can be drawn about these various things from a statement is a separate matter.)

      Anyway, what I pointed to in my post was statements about how somebody came to be enthusiastic about the activity, and, personally, that story for me is the least useful part of such statements, even though it is very commonly how such statements start.

      I don’t agree with the advice Graham received to “keep it about research I had done and potential fit.” Regardless of whatever might be ideal, my own experience is that all manner of other matters can be quite influential to some evaluators. But obviously different disciplines differ and different departments within a discipline differ.

      Like

      1. I agree with Jeremy’s list and I did narrow in on an aspect of soc SOPs that I don’t really have direct experience with (though that’s the trouble with “scientific” discussions on non-quantifiable admissions criteria generally).

        I carried my advice from economics where getting personal in the personal statement earns a negative evaluation, and candidates are instructed to make it as bland as possible and let their writers speak for them. Economics has more competition and a more efficient marketplace at the grad and post grad levels, so they don’t email with candidates pre-admission, want to hear cute ideas in the SOP, or life stories.

        Moreover, I considered it to my advantage to conceal my politics and opinions as much as possible to avoid discrimination. I got admits exclusively to the less-Marxist econ soc-y departments and quanty departments, so I think that was a good strategy.

        There is a document floating around out there written by a team of psychologists instructing people on SOPs, and they explicitly tell people not to include their personal struggles with mental illness. It’s considered unprofessional and embarrassing because empathy does not imply insight.

        Then again, psychologists don’t have standpoint theory, a rat’s nest of contradictions that says you’re *more* insightful if you’re a Simmelian stranger studying dominant groups (e.g., a minority studying whites), but that you’re *less* insightful if you’re a Simmelian stranger studying oppressed groups (Alice Goffman studying blacks).

        Like

  3. “We found 53 [of 88] responses [sampled from DGS’s at 457 programs] related to damaging personal statements, which we sorted into four subcategories: personal mental health, excessive altruism, excessive self-disclosure,
    and professional inappropriateness.”

    http://psychology.unl.edu/psichi/Graduate_School_Application_Kisses_of_Death.pdf

    I’m not playing a superiority card here. I wrote an undergraduate transfer application to Chicago that included an essay, with quoted lyrics, about how reggae and hip hop made me more socially conscious, and how drug dealing spoke to my ambition and intelligence. Application denied.

    Like

  4. “One applicant admitted to feeling ‘a thrill of excitement every time he/she steps into a morgue.’ Another wrote ‘a 10-page narrative of herself as Dorothy on the yellow-brick road to graduate school.’ A third indicated that he or she ‘had performed (acted?) in pornographic movies, which was not well received by the admissions department in consideration for acceptance into graduate school.’ Other types of professionally unsuitable content include using excessive or inappropriate humor, ‘cutesy/clever stuff,’ and excessively religious references (e.g., ‘I am a gifted therapist naturally. God has given me natural talents that make me a very good clinician. This was recently demonstrated when I helped my devil-worshipping brother go on the right path, God’s path.’).”

    (I also made some reference to “beating an egg like it owes me money” in an “other skills” section of the UG Chicago application where I explained how to make great scrambled eggs.)

    Like

  5. Very clever Jeremy. Transferring the conclusions to the other post, which potential grad students will succeed is largely driven by desire (or whatever the third category is) after requiring a certain level of ability and skills. I’m generally persuaded. It helps to think about three dimensions and this is a good example of how formalization/simulation can be clarifying. That said, a couple of questions:

    1) what does relative scarcity among the three categories (ability, skill and desire) do to your predictions? That is, if there is an uneven distribution of availability of candidates with sufficient ability, skill, desire, does that change the implications of your results? For example, if we had 40 applicants to our grad program that had sufficient ability and skill, but only 10 with sufficient desire (some among 40 and some not), would the scarcity of desire mean we should select on desire regardless of skill and ability?

    2) could you estimate at what precise point skill and ability become orthogonal/irrelevant to predicting success? Or, the relative weight we should attach to desire as we move up the ability and skill continuum? Would the correct conclusion be that the person has to have extremely high desire to attain success if ability and skill are below some level?

    3) what if ability, skill and desire each have differing levels of measurement error?

    4) hardest part: Given the distribution of GRE scores among soc-intenders that I linked to earlier, could we estimate at what level we can safely ignore GREs? I know the following are unknowns in my example (success, desire, the correlation b/w GRE/desire and success), but one could assume a distribution of correlations and come out with some probabilities, right? If you are a moneyball DGS of a sociology program, it seems this is the strategic issue.

    Not saying you’re going to do all this – though happy to see results if you do! – but just curious if you think these implications are on the right track.

    Like

    1. I don’t know if I have any broader normative points beyond the problem of seeing an association (or non-association) within a selected group and failing to think about how that association was influenced by how the group was selected in the first place.

      As for #3, though: I could simulate up the implications, but one could take “skill” in this example and imagine it was measured with error. Right now, “skill” is measured with no error, and behaves like “height”. If the “skill” measure was nothing but error, then you’d be in the position the model is right now with “desire.” Anything in-between would be in-between. So the worse you measure something, the more it will be associated with differentiating people in the selected group, because the less it was available for use in selecting people into the group to begin with.

      Liked by 2 people

    2. In Jeremy’s model, the relationship between skill/ability and success (in the whole sample) is linear and therefore greater skill/ability continues increasing success even after high levels are achieved. So regarding your (2), you can assume (or argue) that the relationship is non-linear with declining marginal benefits to skill/ability, but it’s not obvious that this is true. The weak correlation between skill/ability and success in the selected sample might make you think that there are declining marginal benefits to skill/ability, but the point of Jeremy’s model is that this weak correlation IS NOT good evidence for that hypothesis.

      Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s