When should you make a complaint?

I’m writing this post with some trepidation, as dealing with complaints is the most difficult part of being an editor. However, I think there is much to be gained from bringing some clarity to the process and by giving a few broad guidelines on when it’s appropriate to challenge a decision.
First, some general suggestions that will make it more likely that any challenge will be successful:
1) Never reply immediately to a decision letter you didn’t like, as you’re almost certain to write something you will regret later.
2) Don’t use ‘Reply All’ to pass the decision to your co-authors, especially if you want to include some invective against the editors or editorial process, as these sometimes end up coming back to the journal.
3) If you have a hunch that you know who the most negative reviewers are, forget about it. Claims that Dr X was probably ref. 3 and that he’s always hated your work sound ridiculous, and also constitute a fairly strong ad hominem attack on Dr X’s integrity. Furthermore, guesses of this sort are wrong most of the time.
4) Even if your nemesis was ref 3, and they gave a signed negative review, this isn’t grounds to overturn the decision. The editor was almost certainly aware of this issue when they decided to reject your paper, and the rejection implies that the editor either agrees with Dr X’s criticisms of your work or else based the rejection on comments of another referee.
5) If you’re convinced that the decision should be challenged, check with your co-authors. They may be more circumspect and may prefer to submit elsewhere rather than getting into a scrap with the journal.
Once you’ve calmed down and taken a few days to read through the decision, and you’re still convinced that it’s not fair, then compose a polite and carefully worded letter that explains why you think the decision needs to be revisited. To save everyone a bit of time, challenging the decisions along the following lines almost never works out:
– “you gave my paper a ‘reject, encourage resubmission’ decision, but it’s better than that and it should have been an acceptance with minor revisions”. Them’s the breaks. You’ve got a second chance, so do your best to make sure that the resubmitted version is brilliant. For Molecular Ecology, a reject-encourage typically indicates that the paper needs extensive changes (with potentially additional data) and that the editor wants to send it through the review process again. It’s generally best to respond to individual reviewer and editor comments as part of the resubmission rather than to challenge the decision itself.
– “two of the reviewers loved it and only suggested minor edits, but nasty referee 3 made you reject it”. A decision isn’t based on an average of the review recommendations, and if the comments of ref 3 convince the editor that the paper is flawed, no amount of adulation from reviewers who missed these problems is going to change that.
– “I accidentally included the wrong dataset/analyses/conclusions, and this is why the paper was rejected”. We do our best to employ psychic reviewers who are able to access the ‘true’ version of your manuscript through the ether, but these are increasingly hard to find. Please read through your paper before submitting it.
– “you published a very similar paper back in 1996, so it is inconsistent to reject our manuscript now”. The field is constantly changing and the standards for publication are always moving higher. If your data and methods are de rigeur for five or even ten years ago, it saves everyone a bit of time if you submit to a more appropriate journal. Furthermore, even if a paper similar to your ms has appeared in the most recent issue, it’s likely that the accepted paper has other merits that led to its acceptance. It’s also likely that we’ve rejected several other papers like yours already this year, but of course this info isn’t public.
– “we resubmitted our paper, but while the original reviewers liked the new version, the comments you received from the new referees made you reject it”. Molecular Ecology has a policy of only allowing one ‘reject, encourage resubmission’ per paper, and this is very effective in preventing interminable rounds of ‘resubmission/finding new reviewers/they find new problems/yet another reject-encourage’. Resubmitted papers do get accepted about 70% of the time, so clearly it is possible to prepare a new version that convinces both the original and new reviewers that it’s worth publishing. Lastly, if none of the original reviewers were able to look at the resubmission, we have to use new referees.
So when should you challenge a rejection? The reason that most often leads us to revisit a decision is evidence that the peer review process was flawed in some way that unfairly biased the review process against your paper. For example, a reviewer might claim that a particular analysis is vital to your paper. However, if the analysis is actually in the paper and somehow neither the editor or the reviewer noticed it, then certainly the decision would be revisited. Similarly, if a reviewer convinces the editor that some aspect of the methods is fundamentally flawed, but the criticism can be shown to be incorrect, then it would be worth appealing the decision.
It should be noted that while instances like the two described above would be grounds for appeal, there is no guarantee that the decision will be overturned. It might be, for example, that there were other major flaws in the paper that were largely responsible for its rejection rather than the mistakes made by the referees and/or editors.
I don’t want to make it appear that we never overturn decisions. We do when we are convinced by the authors that a mistake was made in the review process resulting in a faulty decision. This is infrequent, about one in every 50 rejections, but it does happen!