on sharing work in progress and anonymity

I got involved in a debate over at orgtheory about the pluses and minuses of putting working papers on line at SocArXiv (or elsewhere). That debate was tangled up with a variety of issues around the proposal to require public posting of papers that win (or are submitted to) section paper award competitions.

In this post I want to avoid that tangle of other issues and open discussion/debate on the narrower question of whether the discipline of sociology as a field should do all it can to move toward the model of other fields, where working papers are routinely placed on public archives before they go through peer review for ultimate publication.

The sociology model as it is generally practiced involves writing a paper, presenting it at conferences and circulating drafts of it around for a year or more, submitting it to a journal, going through several iterations of rejections and R&Rs, and finally getting it published maybe 4 or 5 years after it the work was originally done. In the meantime, some people (those you were at conferences with or to whom you sent the paper) know about the work, while others working in the same area may not know about it and thus will not cite it or be influenced by it, junior scholars worry that their work will be scooped by a more senior person who gets the idea from a circulating PDF or as an anonymous reviewer, and knowledge as a whole bogs down.

The alternative model practiced in many fields is: (1) Do the work and present it at conferences as the work evolves.  Be known as the person/team working on problem X because you have talked about it at multiple conferences. (2) Post a working paper on ArXiv or SSRN etc. as soon as you think  you have something to report. (3) Other people cite and debate your work based on the ArXiv or SSRN etc version. If it is wrong it gets called out and fixed. If it is novel and correct, you get invited to more conferences to discuss it and you learn about the work others are doing in the same field. (4) Your paper slogs its way through peer review and ultimately gets published; then you link to the published version from the working paper site. 

I advocate moving from our current model to the open model. I think the academic field as a whole wins when the work is made public and accessible as soon as possible: the author wins from getting their work noticed, and knowledge wins from everybody knowing about it. This is also the best model for influencing public debate outside the academy. I think it is also best for young scholars. The best way to have some protection against others plagiarizing your work (or some recourse if they do) is to be able to prove when you did the work, and the best way to do that is with a time-stamped public archive. Additionally, it allows young scholars to get known while their work is still making its way to peer review and publication.

Changing models does involve some real issues. Here are the important issues and my thoughts about them:

(1) Issue: Posting work too early will make me look stupid, as the work will improve over time. Alternate version: other people posting work too early gives me too much bad work to read. Response: Don’t post work you think is stupid. But as the work improves you can: a) replace the paper with an updated/corrected version; b) make the paper unavailable if you realize it is wrong (i.e. basically retract it). My own evolving strategy has been to use the abstract on a working paper to state that this is a paper in process subject to revision. In one case, when I found an error in the paper, I annotated the corrected upload to note that it corrected an error in the earlier version. As far as other people’s work, you and I know that there is already a lot of published work that isn’t worth reading. Do what you do now: look at abstracts, skim possibly useful pieces, read closely only the ones that seem worth your time.

(2) Issue: Posting the work publicly will let other people know my name! I won’t be anonymous. Response: Yes, that is exactly the point. How else do you expect to build a career? Response to response: But what if Important People get mad at me or try to hurt me because I criticized them or said something that is politically unpopular? Or think my paper is stupid or wrong and look down on me? Response to response to response: These are real issues. If you write something with your name on it, your are held accountable for it. That is the price you pay if you want to be a scholar. Science accumulates from known individuals who stake their personal reputations on the veracity of what they write. You can be fired from a tenured professorship for falsifying your research reports. Fiction writers can be anonymous, but academics cannot. Anonymous blogs and social media accounts are another way to express opinions without being accountable (although it is getting harder and harder to stay anonymous).

(3) Issue: A standard practice of posting working papers before publication means that peer review will not be blind to the name of the author. Response: This is the biggest institutional change and needs some serious discussion. Does our current practice of removing the author’s name from publications undergoing peer review add enough value to outweigh the advantages of public posting of working papers? Does double-blind reviewing eliminate or at least mitigate the known biases against women, minorities, junior scholars?  This is a complicated issue. First, blinding the author’s name is already often a charade, a ritual practice in which reviewers pretend that they do not know who the author is, when they actually do. Competent reviewers will automatically recognize the work of the key people in their field and many reviewers Google a paper they are asked to review. (For the record, my own practice is to wait to Google until after I have written the review.) Additionally, the editors of journals, who are the ones who actually make the decisions, are never blind to the author. Most of us believe that the integrity of the review process depends upon the anonymity of the peer reviewer, as any journal editor or frequent reviewer will tell you that people are quite willing to criticize their friends under the cloak of anonymity. Most people who review value their integrity as scholars too much to lie about what they think about a paper. At the same time, the matter of conscious or unconscious discrimination by reviewers should not be rejected out of hand. I do think there is symbolic value in leaving the author’s name off the paper because it signals that their status, gender etc. ought not to be a factor. The forms that reviewers fill out could also include questions: Have you read this paper prior to being asked to review it? Do you know who the author is? Do you suspect who the author is? Is the author a friend? An enemy?

(4) Issue: Publishing working papers destroys the value of the peer-reviewed journals. And what about books? Publishers cannot afford to print materials that people can get for free on the Internet. Response: This is where we dig into the core of the open science debate. The current model is that universities pay twice for science, first by paying the professors and other staff to do the research, and again by paying publishers so they can put the work in a library.  Private publishers, who did not found journals nor do the entrepreneurial and academic work to build their reputations and prestige, have purchased the rights to journals and are extracting rents from them. Professional associations have taken a cut of these rents and so work to protect the publishers. This is the core of the fight. I will note that math and physics both rely on the ArXiv model (working paper before publication) and both fields still have private publishers extracting rents from them, so it is not clear that posting working papers actually hurts publishers, as long as the peer-reviewed article remains the gold standard for tenure.

The economics of the book publishing business is different and I am less certain how to address the book issues. It has become the case that many PhDs now embargo their dissertations and do not send them to Dissertation Abstracts as part of protecting the publishability of the book. This seems bad to me, as the record of dissertations (who did them, when, where, under which advisor) is itself of scholarly value. I will leave it to more knowledgeable people to debate the book business.

UPDATE: An additional issue: A journal will not accept my article if it has previously been posted on line. Response: This is one of the ongoing issues in open science. Journals vary in their policies, and disciplines vary in the mix of journal policies. I’ve heard that biology  and medical science are more closed, for example. Most sociology journals do not consider this to be prior publication, but there certainly is a risk that the defense of journal rents could go down this path. This is a link to a site that tells you the policies of specific journals. I think I saw a blog somewhere that pulled the sociology journals from this list, but I don’t have the link right now. It gets complicated because federal law increasingly requires that scientists who have received funding put their results in an open access place, so we are all facing pressures that go in different directions. In general, most sociology journals treat the paper you sent to them before review as yours to do what you want to with, and only restrict your use of the copy of the published version; some even let you post that.

 

Author: olderwoman

I'm a sociology professor but not only a sociology professor. I keep my name out of this blog because I don't want my name associated with it in a Google search. Although I never write anything in a public forum like a blog that I'd be ashamed to have associated with my name (and you shouldn't either), it is illegal for me to use my position as a public employee to advance my religious or political views, and the pseudonym helps to preserve the distinction between my public and private identities. The pseudonym also helps to protect the people I may write about in describing public or semi-public events I've been involved with. You can read about my academic work on my academic blog http://www.ssc.wisc.edu/soc/racepoliticsjustice/ --Pam Oliver

9 thoughts on “on sharing work in progress and anonymity”

  1. This is wonderful, OW, thank you.

    On the blind system. We should also consider that the current system enables the bad behavior of senior people who are protecting their research turf. I think this happens all the time, at different levels of subtlety. If I am known in a field, I am likely to be asked to review work in that area; I am also likely to have an interest in protecting my turf, maybe because I or my students have new work in the pipeline, or because I just don’t want to be superseded by new approaches (I can tell you from experience, this hurts my feelings), or because I am just biased toward my way of doing it. The nice person in my position may cast an unintentionally critical eye on the paper under review, and harsh on it. The evil person in my position simply finds a way to reject it — and they do it with impunity, protected by anonymity. From my observation, I think this is more prevalent and damaging than the discriminatory treatment of junior scholars — or minorities or women — whose status is identifiable to the reviewer. I think the more prevalent bad behavior is about personal self-interest than about group discrimination, and it results more from hiding the identify of reviewers than from protecting the identity of authors.

    It’s easy to see possible problems with opening peer review, but those need to be compared with a realistic assessment of the problems with the current system, not with it’s idealized self-image.

    Liked by 1 person

    1. Hmm. Regarding turf protection, it does seem that the open science model also addresses this, as it allows people to make it clear that they are working in an area without being blocked by self-interested gatekeepers.

      Liked by 1 person

      1. I’m against naming reviewers because I think it starts opening up a quid pro quo that should not be there, but adding questions to a reviewer report that specific ask about conflicts of interest in terms of do you have work in progress in this same area seem more likely to help. Editors can help police this if they are told it is part of their job.

        Like

  2. For your consideration: https://f1000research.com/

    Immediate Publication: The traditional anonymous pre-publication peer review of research articles can cause long delays before new results become visible. F1000Research uses an author-led process, publishing all scientific research within a few days. Open, invited peer review of articles is conducted after publication, focusing on scientific soundness rather than novelty or impact.

    Transparent Refereeing of Articles: Open peer review of articles removes the secrecy and anonymity that can bias the way scientists critique each others’ work. In F1000Research, signed referee reviews and author responses are published alongside each article. Authors can publish revised versions of their articles at no extra cost. All articles that pass peer review are indexed in PubMed.

    Like

  3. This is terrific – thanks for it. I will offer one piece of somewhat self-interested data in response to this:
    “And what about books? Publishers cannot afford to print materials that people can get for free on the Internet.”
    The MIT Press, oversight of which is part of my purview as Director of MIT Libraries, published Deep Learning (https://mitpress.mit.edu/books/deep-learning) as free online version and a reasonably priced print version, and the print version is a best seller. OA online and reasonably priced print versions can co-exist.

    Liked by 2 people

  4. I like the strategy of switching blogs to refocus the conversation. Thanks for this very clear overview. To my mind, the benefits of shifting toward active discussion of working papers far outweigh the costs. That said, I do think that the one serious issue is the long-tail risk of going viral in a bad way for people who work on controversial topics. There is a whole little industry devoted to seeking out academic work with controversial or ridiculous sounding abstracts and publicizing it (e.g. Campus Reform). The people who are most likely to be targeted by that are disproportionately minorities of various sorts. However, if someone’s research is going to be attacked, I don’t think it makes much difference if the paper is published or a working paper. And our response shouldn’t be to avoid working papers, but to actively defend scholars who are attacked, as individuals and institutions.

    Also, here’s a nice example of how things should work with working papers. An NBER working paper recently found that physicians trained at top medical schools prescribed opioids less than physicians from lower ranked schools. It got a bunch of press, but its methodology was also quickly challenged: https://www.usnews.com/news/national-news/articles/2017-08-09/experts-question-relation-between-medical-school-and-opioid-prescriptions. Presumably, the authors will revise the paper before submitting to a journal to address the criticisms. But the conversation is happening. As this thread suggests, the uproar shows the success of working papers, not the failure: https://twitter.com/timothyjlayton/status/895374756295454721

    Liked by 1 person

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s