direct and indirect effects of “citation padding”

Phil has had a couple of posts now about the practice of journal editors encouraging citations to a journal that they edit, and it sounds like there may be more.  I should say that I don’t recall ever having an editor say something as… direct as the statement Phil quotes, and I do remember being on projects where, on our own initiative, we’ve inserted references to a journal or the work of its editor with “can’t hurt our chances!” rationale.

One might think the specific practice of editors encouraging citations to their journal for impact-factor purposes could be curbed by simply eliminating journal-level self-citations from impact factor counts.  But: my suspicion is that when people insert citations in with the idea of pleasing editors at a specific journal, they mostly don’t bother to remove those citations if the paper gets rejected from the journal anyway.  In other words, when journals encourage authors to cite other articles in their journal, there’s a direct and readily observed effect on impact factor as self-citations, but then there is also this hidden and downstream effect of papers that are published elsewhere.  Depending on the journal’s acceptance rate and how early in the process references are added, the indirect could potentially be substantial relative to the direct effect.

On the bright–if somewhat perverse side–the practice could actually be good for anybody who wanted to try to use networks of citations across journals to make inferences about journal prestige.  Because if a publication follows a chain of Journal A -> Journal B -> Journal C -> Journal D in order to get published, Journal D will have the traces of efforts to please Journals A, B, and C, including citations to those journals, whereas if it had been accepted by Journal A, it wouldn’t have traces to please B, C, and D because it was never sent there.  Put another way, the order in which authors send articles would be a good way of sussing out the hierarchy of journal prestige, but that’s private information, but authors including gratuitous citations to those journals and then leaving them in is a way in which that private information can be made visible.

another peer-review horror story

Just in time for Hallowe’en, Phil Cohen has posted an account of a recent experience of trying to publish an article. The account is more striking when one pauses to think that the story is not getting told because it is extreme in a discipline-wide sense, but that it’s extreme for one of the few folks who write blog posts about things like this. In other words, too many people with too many papers are ending up with these sort of stories.

I appreciated Phil’s forthrightness in the account, particular the part where he reproduced one editor’s request to insert citations to more papers from their journal.
Beyond that, I was particular fond of this paragraph of the summary:

Sociologists care way too much about framing. Most (or all) of the reviewers were sociologists, and most of what they suggested, complained about, or objected was about the way the paper was “framed,” that is, how we establish the importance of the question and interpret the results. Of course framing is important – it’s why you’re asking your question, and why readers should care (see Mark Granovetter’s note on the rejected version of “the Strength of Weak Ties”). But it takes on elevated importance when we’re scrapping over limited slots in academic journals, so that to get published you have to successfully “frame” your paper as more important than some other poor slob’s.

screwance

Kieran Healy has written a paper about nuance and posted it here.  It’s an argument that resonates with my own experience, especially in terms of various forays of reading efforts of social theory to talk about the relationship between what they are doing and psychology or, worse, “biology.” While there’s various colorful language throughout the paper, this unadorned sentence hit home for me in that regard:

there is a desire to equate calling for a more sophisticated approach to a theoretical problem with actually providing one, and to tie such calls to the alleged sophistication of the people making them.

genes and infidelity

Phil mentioned in the comments of an earlier post the recent news story about how martial infidelity has a heritability of .4, and the news story also features various more specific claims about the specific genes and systems supposedly involved and the purported evolutionary psychology of it all. Eric Turkheimer, who is hopefully already established as Sociology’s Favorite Behavioral Geneticist, has a nice blog post in which he explains problems with the news story. Enjoy!

aside on the heritability of everything

(Substantial prelude with some light technical bits, feel free to jump to [UPSHOT] or [BOLDFACE PUNCHLINE])

As is shown in the meta-analysis Andy references in his last post, more or less every measurable outcome anybody cares about in any study of human beings is more similar among identical twins than it is among fraternal twins, which in the classical model applied to twin study data means the trait has a non-zero heritability.

Perhaps the major motivation of the giant meta-analysis, however, is evaluation of the extent to which identical twin correlations are twice the fraternal correlation.

the imitation game

Quote from Alan Turing that I came across while reading Lee and Wagenmakers’s Bayesian Cognitive Modeling:

“I assume that the reader is familiar with the idea of extra-sensory perception, and the meaning of the four items of it, viz. telepathy, clairvoyance, precognition and psycho-kinesis. These disturbing phenomena seem to deny all our usual scientific ideas. How we should like to discredit them! Unfortunately the statistical evidence, at least for telepathy, is overwhelming.”

assessing james coleman

From Sharon McGrayne’s The Theory That Would Not Die, about Bayesian statistics (versus frequentism):

The chasm between the two schools of statistics crystallized for [Howard] Raiffa when Columbia professors discussed a sociology student named James Coleman. During his oral examination Coleman seemed “confused and fuzzy . . . clearly not of Ph.D. quality.” But his professors were adamant that he was otherwise dazzling. Using his new Bayesian perspective, Raiffa argued that the department’s prior opinion of the candidate’s qualities was so positive that a one-hour exam should not substantially alter their views. Pass him, Raiffa urged. Coleman became such an influential sociologist that he appeared on both the cover of Newsweek and page one of the New York Times.

Bonus for those interested in standardized tests: How did Raiffa end up as in a position to evaluate James Coleman?  The same book tells his academic origin story:

one book, one _____

My favorite contemporary novel, The Amazing Adventures of Kavalier and Clay, is the selection for One Book, One Chicago. I was surprised when I heard this, because I thought, “But… it’s a novel.”

I realized that I’d just assumed that the trajectory of One Book, One Northwestern was a general phenomenon, where the history has been:

2014-5: Whistling Vivalidi, Claude Steele (non-fiction)
2013-4: Last Hunger Season, Roger Thurow (non-fiction)
2012-3: Never a City So Real, Alex Kotlowitz (non-fiction)
2011-2: Immortal Life of Henrietta Lacks, Rebecca Skloot (non-fiction)
2010-1: Mountains Beyond Mountains, Tracy Kidder (non-fiction)
2009-10: Hot, Flat, and Crowded, Thomas Friedman (non-fiction)
2008-9: The Reluctant Mr. Darwin, David Quarmmen (non-fiction)
2007-8: Go Tell it on the Mountain, James Baldwin (novel)
2006-7: Othello, Shakespeare (play)
2005-6, Antigone, Sophocles (play)

Most obviously, a sharp movement away from literature. Do other universities have a One Book program? Has their trajectory been like Northwestern’s, or have they remained more literary-focused?

the personal touch

I got a video message from Northwestern’s football coach about renewing my season tickets:

http://www.nusports.com/video/Fitz_message_090.html

The fun part is, if you change the 090 in the URL, you can see the greetings to other first names. As of now, it goes up to 108. He certainly does a better job than I would at being engaged through each of these different messages.

how long should asa papers be allowed to be?

My last post noted that ASA changed its rule from having a 20 page limit to having now saying 15-35 pages. I think this is a good change. Fabio dissents:

Dissent here: I oppose paper bloat. Thus, I always praised the 20 page limit because it forced people to get to the point. You only have 15 minutes to present, for crying out loud.

My view on this is that I would never write a paper just so I can present at ASA, and I wouldn’t advise anyone else to, either, except maybe people who are in positions where they have no interest in publishing but are really desperate for travel funds. In my preferred world, ASA would allow people to submit slide decks, as well as something much more promissory than a full paper if it’s going to be due over 7 months before the conference. Specifying “working/draft paper” is definitely progress.

In the end, ASA does require papers. In that world, I’m for whatever minimizes the extent to which people end up doing work that diverges from the paper they are writing for publication, simply for the purpose of pleasing the one person that will end up looking at the abstract and possibly skimming the paper (i.e., the ASA session organizer). The problem with the 20-page rule is naive folks would spend irrational amounts of time cutting or revising longer papers in order to meet the page limit. This makes no sense.

asa rule change?

ASA rule for submissions:

15-35 page draft paper/working paper either converted to a PDF file or prepared as a Word, or WordPerfect document stored locally and ready for uploading.

This is new, right? Didn’t it used to be 20 pages, without anything to imply it shouldn’t be a fully complete paper?

My preferred belief is that this is a new change and that, regardless of whatever story exists about its provenance, blog & Twitter whinging helped bring it about (e.g., here). But I’m open to counter-facts and counter-narratives.

In any event, regardless of why, kudos to the organization for the change.

beckieball and the study of not-quite-elite selected groups

Beckieball is a new sport sweeping the country. Beckieball performance is the result of three things: height, skills, and desire, all of which are uncorrelated with one another. Let’s fire up Stata and show what the correlation matrix would look like for the population:

             |  perform   height   skills   desire
-------------+------------------------------------
     perform |   1.0000
      height |   0.5751   1.0000
      skills |   0.5751  -0.0072   1.0000
      desire |   0.5774  -0.0000  -0.0005   1.0000

Unfortunately, scouts cannot really assess desire well, so when they are picking players for professional beckieball leagues, their assessment is just based on equal parts height and skills. What happens?

that nba example

OK, so the NBA analogy I invoked the other day has prompted some confusion. I’m in Australia and have two ASA papers to finish and several more room-escape-puzzles to try, so I don’t have time for a longish post about this, but:

great moments in graduate advising

From Diederik Stapel’s memoir:

When students gave presentations of research that I knew was based on my fake data, I could barely wait for them to finish. Maybe someone would burst the bubble, pointing out a fundamental error that would show that the whole thing was completely fictional. I sat and watched a great show, but I was the only one there who knew that it was just an illusion. I watched students, doctoral candidates, and research assistants talking enthusiastically about their work, and I knew that the data they were presenting were all fake. It was hard to keep up the pretence. What they were presenting was fantastic, as coherent as anyone could hope for. They had the results they’d hoped for, they’d passed the test, they were full of self-confidence, they believed in themselves and in the reliable structure of science, but I knew that their faith was built on sand.

selection

Among NBA players, there’s very little correlation between a player’s height and how many points they score. Michael Jordan was slightly below average height for an NBA player; Allen Iverson was in the bottom decile. And yet no one would argue the height is irrelevant for someone’s prospects at playing pro basketball: the average NBA player is nearly 6’7″, that is, in the top percentile of the male population.

The relationship between GRE and graduate outcomes is like this. If you take a group that has already been selected on a number of characteristics, including test scores, then there is often only a modest correlation between test scores and outcomes. But exactly the same could be said for any of the other criteria that were used, like GPA, or like the quality of the writing sample or the ever-amorphous assessment of “fit”, if either of the latter were to be assigned scores.

One can not extrapolate from this reasoning to draw conclusions about how people should be selected in the first place. If you did, you’d conclude that nothing departments use to admit students matters much for whether or not they will succeed, and that perhaps the fairest thing to do would just be to admit students completely at random. Admittedly, this would save departments a lot of time.