Tim Vines is an evolutionary ecologist who found his calling in the process of peer review. He was Managing Editor of Molecular Ecology from 2008 to 2015, launched The Molecular Ecologist in 2010, and is the founder and Managing Editor of Axios Review. Here, Tim is The Fourth Reviewer, taking on your questions about peer review and publishing. Got a question for the Fourth Reviewer? Send us an e-mail!
I’ve seen you take numerous pot shots at open peer review. What gives?
I have to admit that I’m not a believer in open peer review. It’s always seemed like the potential downsides of the solution outweigh of the problems it purportedly addresses. To be clear, I’m talking about peer review where the reviewers know the identities of the authors, and the authors are given the identities of the reviewers. Signed review, where the names of the reviewers are made public alongside the accepted article, is an entirely different beast and won’t be discussed here.
There are three critical issues that I see with open peer review:
1) The potential for revenge against critical reviewers. This quote in a piece about open peer review by Jon Tennant particularly struck me:
Early career researchers are perhaps the most conservative in this arena as they may feel afraid that by signing overly critical reviews (i.e., those which investigate the research more thoroughly), they will become targets for retaliatory backlashes from senior figures. The traditional double blind process therefore offers, in theory, a sort of protection for those in junior positions. …
In a perfect world, we would expect that strong, honest and constructive feedback would be well received by senior researchers, but there is an apprehension that this would not be the case. However, retaliations to referees in such a negative manner are serious cases of academic misconduct, and likely to be dealt with as such.
Unfortunately, scientists are human beings. There will always be power asymmetries between them, and everyone will at times have the opportunity and desire to take revenge on others they feel have given them unfair criticism. Since this revenge would take the form of rating an enemy’s top-quality manuscript or grant proposal as ‘not great’, there is just no way that it could be detected, let alone result in disciplinary action.
Let’s assume (generously) that only 1 in 10 reviews is perceived by the author to be sufficiently wrong/over-harsh/sloppy that the reviewer is deemed an idiot or an enemy. Assuming 2.5 reviewers per paper, a researcher reviewing 20 papers/yr and publishing 10 would add 2.5 of their colleagues to their idiot/enemy list each year and be thought an idiot/enemy by 5 other researchers. Academics have excellent memories for this sort of thing (after all, the main currency of academia is reputation), so these idiot/enemy lists would build year on year, and peer review would become ever more distorted by personal vendettas.
2) Criticism is discouraged. The worse the paper is, the more important it is that the reviewer is critical. In blinded review they are able to do this without repercussion from the authors, but in signed review they face the choice of a) be critical and risk making an enemy, or b) be nice, avoid a conflict, but let a bad paper into the literature. Open review thus inhibits the proper review of papers, which is (for me) a fundamental flaw in this approach. Incidentally, this may be why open review often elicits a positive response from authors – it’s set up to be a love-in. Unfortunately, making authors feel good about themselves is not what science or science publishing is about.
3) Where’s the need? It’s unclear (to me) which problems open peer review addresses that cannot be dealt with by improvements to the blinded system.
I’ve heard three broad arguments in favour of open peer review:
a) The transparency of open review prevents conflicts of interest from affecting the review process.
In blinded review, the editor knows exactly who the reviewers are, and is typically aware of potential conflicts. They can then weight reviews from competitors accordingly. To guard against the possibility that the editor is unaware of the relevant conflicts, authors can list the relevant people as ‘non-preferred’ reviewers and provide a detailed explanation in their cover letter.
What extra defence against unfair review does open review provide? The authors could complain that a ‘reject’ decision based on fatal comments from an identified competitor was unfair, but either the editor knew about the conflicts and thought the reviewer was right, or the reviewer was being unfair and the authors forgot to list them as non-preferred. Given that the latter is relatively rare and can be remedied by the journal revisiting the decision, I just can’t see the need here for open review, particularly given the severe problems outlined in (1) and (2).
b) Open peer review reduces abusive or unpleasant comments by the reviewers.
This might well be the case, but I think the prevalence of this issue is greatly exaggerated. From about 25,000 reviews I looked over while at Molecular Ecology, I estimate that I asked at most 20 reviewers to remove ad hominem statements or otherwise tone down their language. I’m sure plenty of Molecular Ecology authors have been upset by the comments on their paper, but as far as I could tell, these comments were objective and focused on the manuscript (rather than the authors). Journals screen reviews for offensive comments, so most ‘abuse’ should never reach the authors. More broadly, the goal of peer review is to ensure that the published literature is as robust and error-free as possible: if you find the process too soul-crushing either a) write better papers, or b) find a job where nobody criticises your efforts.
c) Identified reviewers are less likely to submit sloppy or erroneous comments.
This is really a corollary of (b), in that open review is expected to improve the overall quality of reviews. I think this idea partly stems from the misperception that the reviewer is anonymous to everyone in blinded peer review, and can therefore say whatever they like without repercussion. In fact, the editor always knows the identity of the reviewer. Submitting shoddy reviews imperils the reviewer’s reputation, and since the editor is typically a respected researcher in the reviewer’s exact field, that can have substantial negative consequences. How much more of an review-improving effect can we expect from sharing the reviewer’s name with the authors as well?
Some supporters maintain that they do find open reviews to be higher quality, but if this comes from the authors’ standpoint they may just be conflating review quality with whether or not the reviewer liked their paper. In support of this, one of the few randomised controlled trials on open review (van Rooyen et al (1999)) found that editors rated reviews that either rejected or accepted the paper as similar in quality, but the authors felt that the ‘reject’ reviews were much worse and the ‘accept’ reviews better. The same paper also found that anonymous reviewers rejected 8% more papers than identified reviewers, although this comparison was not significant (N = 114). It would be really valuable to repeat this study across more journals, as in the absence of hard data we’re reduced to unsatisfying verbal arguments (like the above!).
Open peer review has many admirers, and some of its most active advocates are true experts in this area. It may be a field-specific issue, in that the case for open review may be much stronger in medicine than in evolutionary biology. Or maybe I just haven’t fully understood the arguments in favour. Either way, I’m happy to be convinced that open review really is better than single or double blind – it just hasn’t happened yet.
