Haldane's Sieve

This week we have a guest post by Graham Coop and Joe Pickrell. Here, Graham [GC] and Joe [JKP] answer a few questions we had about the development and future of their blog, Haldane’s Sieve. If you’re interested in population genetics it’s definitely worth your time to take a look at and follow the blog. We would also like to hear what the readers of the The Molecular Ecologist think of Haldane’s Sieve and its place in the research community, so feel free to post in the comments section below.
1. Can you explain what Haldane’s Sieve is, and what service it provides to the science community?
Haldane’s Sieve is a site designed for the promotion and discussion of preprints in evolutionary genetics and genomics. Each day we post abstracts and links to the preprints that have appeared on the arXiv & bioRxiv and fall broadly in evolutionary genomics. We also invite authors to write posts giving informal summaries of their work,  e.g. explaining the story behind the paper or highlighting some part of the work that they find particularly exciting. All papers also have comment threads where readers can give feedback, and we also have the occasional post with a detailed review of a paper.
[GC] It’s been really great to see so many people in the community embracing preprints over the past few years, and evolutionary genetics and genomics being central to the move towards preprints in biology. It’s great that Haldane’s sieve has played some role in that, especially as it started from a idle chat on a saturday afternoon that hit on a great name for a preprint website. It’s also been wonderful to see a number of biology journals change towards preprint friendly policies in response to authors really wanting to engage with preprints.
2. What are your views on the pros and cons of pre-print availability and the resultant comments/feedback? How does this compare with the peer review process for most journal submissions?  
[GC] The great pro. of preprints is early feedback on your work from the community. We’ve had some great feedback on many of our lab preprints, with some people writing review length and quality comments (admittedly others have garnered little feedback, but that’s ok too). Your work can be part of the conversation in the field sooner, and others can build on that work and cite it sooner.
It also removes part of the stress associated with the highly stochastic nature of journal review. Sometimes the review process from submission can go fast, other times it can take a year or more. Having the work out early in a citable form is important for early career scientists. If you are on the job market the time it takes papers appear can be very frustrating, especially if the big paper from your postdoc is still pending publication. While no one is going to think that a preprint calls the same weight as a published paper on a job app, it could help to have that work out in public and being discussed.
[JKP] I completely agree about the stress of journal review. Taking criticism is already hard enough, but if a critical review means 6 more months before anyone can see your paper (as it does if you haven’t posted a preprint), the frustration is magnified 100-fold, especially if you think the criticism is unfair. But if a preprint is out and citeable (and potentially already being discussed in the field), that situation is considerably less stressful.
[GC] The biggest downside that people seem to perceive is that their work might be scooped if it appears on a preprint server. It’s not clear that this is actually a realistic concern, if your paper is far enough along that you are willing to put a preprint up, it seems highly unlikely that someone could somehow take the work and publish it as their own. Also your work is citable as a preprint so it is hard to see how you can count this as being “scooped”. Indeed the fact that it is public makes it harder to do that, as reviewers will have seen the preprint and be able to point to the prior work. One of our reasons for starting Haldane’s sieve was to ensure that everyone was aware of preprints to guard against this perception.  One minor way in which this “scooping” concern may be valid is that if two groups are working on a similar topic, then one may be influenced by the other and fail to cite the other preprint for their insights. However, this problem is far from unique to preprints and obviously can happen with published papers failing to cite each other.
[JKP] I haven’t really seen much downside to posting preprints. It’s true that if a group working on a similar topic sees that you’ve posted a preprint, they can rush to get their work out. But that’s also true if you give a presentation at a major conference, so personally that doesn’t bother me so much–if I’m ready to make my work public in some way, I’d prefer to get it out to the whole world.
3. What would be your ideal publishing situation?
[JKP] I see the current publication system as a “private, downvote-based” system–people try to publish their most exciting work in the places they’ll get the most readers and recognition, and then get bumped down the scale of visibility via private editorial rejection and occasionally private peer review. I’d prefer a “public, upvote-based system”, where all papers are published as soon as the authors feel they are ready (on an arXiv/bioRxiv-like system), and then papers that get the interest of a community are bumped up to sites that aggregate the most interesting recent work from that community, and potentially after public discussion and review, bumped up again to sites that aggregate the work of most interest to a more general audience. In a system like this, I could follow the bleeding edge work in fields I’m most interested in without delay, while also following the most exciting work from other fields that I follow less closely (where “exciting” is judged by that community) after a delay while that community sorts out what it considers to be truly important advances.
[GC] I mostly agree with Joe. I agree that we should mainly let bad papers sink without a lot of comment. Huge amounts of effort goes into reviewing papers that make there way down the chain of journals, while as a community we could be focusing on publishing, recognizing, & giving comments & feedback, on the papers that move the field forward.
The difficulty really is going to be how we get to a better system. While there’s lots to complain about the current system we need to be realistic about the drawbacks of other potential systems. One problem is getting such an upvoting system to be something other than simply a popularity contest. One of the potentially best things about the current system is that journals try to give everyone a fair hearing (obviously this often fails to be true in practice). With an upvoting system there will need to be mechanisms in place so that the work of junior people or people new to a field gets the attention it deserves. Also while an upvoting system may well be the best system, we need to work out how to ensure that there are ways for dissenting voices, with serious concerns about papers, to get heard.
[JKP] I totally agree about some of the drawbacks, though I’m perhaps more optimistic. It will definitely be important to avoid herd behavior by encouraging dissenting voices and serious criticism; I think this is potentially a huge advantage of the anonymity of the web–in principle, a serious technical criticism of a paper could be raised by a random grad student writing under a pseudonym, while such a person might never even think to put him/herself in the spotlight by going through traditional channels. Honestly, I sometimes read technical “debates” in high-profile journals and think “wow, that argument would be (rightfully) torn to shreds on reddit”.
[GC] I am hopeful that we are moving towards a better system. I see the spread of preprints as one step towards getting us there. Preprints do help to decouple the process of publishing of a paper from the process of peer review and community recognition. That’s a key step, which when combined with new ways of promoting good papers and providing post-publication peer review, could be a great way forward.
4. How do you think something like Haldane’s Sieve will scale in the future as the number of pre-prints and readers increases?
[JKP] A site like Haldane’s Sieve that aims to post all relevant abstracts in a field necessarily has to define the field in such a way that it’s not overwhelmed with content. But as long as the field is well-defined, I think the number of papers published is actually totally manageable–with most papers, I just need a quick glance at the title to know whether or not it’s something I should take a closer look at. For fields that I’m not as familiar with but still find interesting, the types of filters/aggregators I mentioned before will be key–I probably wouldn’t want to see every preprint published on the human microbiome, but I’d appreciate a site where people in that field listed the most important work.
[GC] We are currently debating how we could move Haldane’s sieve forward. An increase in volume as Joe says is not really a concern, and if it ever gets too much we’ll happily be victims of the success of preprints. Rather, we are talking about how we can use the success of Haldane’s sieve toward promoting more community discussion, review of preprints, and working out how to generate a community sieve.