Published 16 March 2016
This resource provides an introduction to and overview of peer-review, the basics of how peer-review and decision making is organized at journals, and the nuances of different peer-review models.
Peer-review is a process used by scholarly journals and edited books to assess the overall quality and importance of the work submitted, and select appropriate material for publication. Some scholarly conferences with an open call for contributions also assess submitted work using peer-review. Traditionally, the goal of the peer-review process is to evaluate a paper in regards to its scientific merit, significance, novelty, and presentation. Peer-review is thus performed by qualified experts that are actively involved in scholarly or scientific research in the same field.
Typically, peer-review is conducted anonymously, meaning that the referees’ identities are not revealed to the authors (single blind peer-review). Some journals use a double blind peer-review model, where additionally the authors’ identities are masked in the paper prior to sending it to the reviewers.
The anonymous nature of peer-review allows for referees to be critical of papers they deem insufficient compared to commonly accepted scientific standards. However, it can also introduce a bias, where referees competing in the same field are dismissing or unnecessarily delaying the publication of scientifically sound papers, only to have the chance to publish similar research results first.
Double blind peer-review aims at removing referee bias that discriminates papers written by certain types of authors (e.g. making distinctions by race, religion, gender, etc.), while encouraging a clear focus on the content and scientific merit of the paper.
Most journals’ peer-review is done before publication of an article and is single blinded. However, due to growing critique of the traditional peer-review process, in recent years, new innovative forms of peer-review have emerged, such as open peer-review, collaborative peer-review and post-publication peer-review:
The journal staff or editors will select qualified experts to be invited for peer-review. Referees are usually selected from the journal‘s editorial board members, from the database of past and volunteer reviewers, by looking up authors of papers cited in the references section, or by searching for authors of related papers in indexing and abstracting databases. Some journals also ask authors to provide the names of a few potential referees, or to indicate persons they want to exclude from peer-reviewing their work. Typically, two referee reports and sometimes more are collected per paper, before an academic editor of the journal will render a decision. Editors may seek additional opinions on a paper, if the first few referees are in strong disagreement, or if one of the review reports is found to be strongly biased or otherwise unusable (e.g. a very short report).
Once enough referee reports have been collected, one or more academic editors of the journal will make a decision regarding the submitted paper. For most journals the decision is solely the editor’s responsibility: the comments by referees are complementary advice and the editor may choose to overrule referees. Other journals may render decisions on papers during a virtual or physical editorial board meeting, or by correspondence. Typically, the journal’s editors will provide a short rationale for the decision when passing the referee reports on to the authors.
Most journals distinguish the following decisions: desk rejection, rejection, revisions with or without additional rounds of peer-review, and acceptance:
Peer-review is an essential system in scholarly communications. Yet, it has some shortcomings:
As a consequence, some argue that the peer-review process is flawed and unnecessarily slows down research. Some journals have thus started open peer-review practices (see next section for types of peer-review). Open peer-review represents a strong mind-shift, and thus many journals and editors have some reservations in adopting this model in fear of scaring away referees in a context where it is already difficult to find qualified referees.
Under the traditional, single blind peer-review model, the referees’ identity is not revealed to authors. However, referees have access to the names of the authors. Traditional, single blind peer-review is conducted before publication of an article: the process leads to the actual acceptance or rejection decision.
Like for traditional, single blind peer-review, the identities of referees are not revealed to authors. Additionally, with double blind peer-review, papers are obfuscated before being sent out for review: authors’ names are deleted from the manuscript. Where authors self-cite their previous papers, language may be altered to mask the identity of authors. Journals that operate double blind peer-review will typically ask authors to submit two versions of their manuscript: the full manuscript, and the manuscript where authors’ identities have been obfuscated.
Double blind peer-review aims at removing a referee bias: referees may not discriminate papers on the basis of race, religion, gender, status of the author, previous controversial papers by the author, etc. Instead, double blind peer-review promotes a focus on content. Double blind peer-review can thus be an interesting model for younger researchers or members of minority groups. Unfortunately, in practice, it is often difficult to fully mask the identities of the authors and, thus, referees can often guess who the authors are.
With collaborative peer-review, authors, referees and editors will discuss and debate the paper in an online forum, before editors render a decision on the paper. Collaborative peer-review is an iterative process between authors and referees, with much shorter and more feedback loops compared to traditional peer-review. Collaborative peer-review has a strong focus on improving papers in a short period of time. Swiss-based open access publisher Frontiers is best known for applying a two phase review system, where the second phase is a collaborative one.
With open peer-review, the records of the peer-review process are published together with the paper, sometimes with the names of the referees. This move helps journals to transparently document the peer-review process for each published paper, and—where referee names are published—forces referees to be considerate. Reports for papers that were rejected are typically not published.
With post-publication peer-review, papers are published first and then commented on by invited referees or the scientific community at large. Examples of journals operating the PPPR model include the open access journals F1000 Research and ScienceOpen Research. Post-publication peer-review is usually done publicly (thus PPPR is also a form of open peer-review). Any paper submitted to these journals will be published first, and referees will then be invited to publicly review the paper. Authors are usually given the chance to revise their paper after a few months, based on the feedback from referees.
However, post-publication peer-review works with any published paper: there are several platforms, such as PubPeer or PubMed Commons, which allow anyone to post comments and discuss published papers. PubPeer’s mission for example is to “help improve the quality of scientific research by enabling innovative approaches for community interaction”. It has attracted more than 35,000, mostly anonymous, comments to date and has become an important vehicle for making allegations of scientific misconduct, according to a Nature News piece published in August 2015.
Version 1 (16 March 2016). This guide was written by Dietrich Rordorf. For questions, please send a message to Dietrich Rordorf, E-mail: firstname.lastname@example.org. Refer to the Resources section on the Ediqo website for more guides and checklists.
© Ediqo LLC (https://www.ediqo.com/), 2016. This article is published under a CC-BY-NC-ND license, which permits reproduction of the entire article, without alteration, for non-commercial purposes.