As scholars and other experts rush to release new research aimed at better understanding the coronavirus pandemic, newsrooms must be more careful than ever in vetting the biomedical studies they choose to cover. One of the first steps journalists should take to gauge the quality of all types of research is answering this important question: Has the paper undergone peer review?
Peer review is a formal process through which researchers evaluate and provide feedback on one another’s work, ideally filtering out flawed and low-quality studies while strengthening others. Academic journals generally do not publish papers that have not survived the process. Researchers often share studies that have not been peer reviewed — usually referred to as working papers or preprints — by posting them to online servers and repositories.
It’s worth noting the world’s largest preprint servers for life sciences — bioRxiv — and health sciences — medRxiv — screen papers for plagiarism and content that is offensive, non-scientific or might pose a health or biosecurity risk. But there are preprint servers in other fields that do not apply the same level of scrutiny.
While peer review is intended for quality control, it is imperfect. For example, reviewers, who often are college faculty with expertise in the same field as the work they are examining, sometimes fail to detect fraud, data discrepancies and other problems. Even some of the most prestigious journals with the most rigorous peer-review processes have had to retract articles. Retractions are rare, however.
“Only about four of every 10,000 papers are now retracted. And although the rate roughly doubled from 2003 to 2009, it has remained level since 2012,” Science magazine reported in 2018.
As of early May 2021, a total of 108 papers about COVID-19, the bulk of which appeared in journals, had been withdrawn, according to Retraction Watch, which maintains an online database of research retractions going back decades.
Despite its flaws, researchers, overall, seem confident in peer review. During a 2019 survey of more than 3,000 researchers across disciplines in multiple countries, 85% agreed or strongly agreed that without peer review, there is no control in scientific communication. The survey — conducted by Elsevier, one of the world’s largest journal publishers, and Sense about Science, a London-based nonprofit promoting public interest in science and evidence — also finds 90% of participating researchers agreed or strongly agreed that peer review improves the quality of research.
Several published studies present similar findings. A 2017 paper in Learned Publishing indicates early career researchers are “generally supportive of peer review” but complain the process is time-consuming and that reviewers, who typically work on a volunteer basis, should be rewarded with some sort of professional acknowledgement or payment.
Regardless of the type of research journalists cover, they should have at least a basic understanding of the peer-review process and its benefits and shortcomings.
Below, we explain some of the most important aspects with help from several experts, including Diane Sullenberger, executive editor of Proceedings of the National Academy of Sciences; Miriam Lewis Sabin, a senior editor at The Lancet; and John Inglis, executive director of Cold Spring Harbor Laboratory Press and co-founder of bioRxiv and medRxiv.
1. Peer reviewers are not fraud detectors. They also do not verify the accuracy of a research study.
The peer-review process is meant to validate research, not verify it. Reviewers typically do not authenticate the study’s data or make sure its authors actually followed the procedures they say they followed to reach their conclusions. Reviewers, sometimes called referees, also do not determine whether findings are correct, given the data and other evidence used to reach them.
Reviewers do examine academic papers to answer a range of relevant questions. They look at whether the research questions are clear, for example, and whether the study’s design, sampling methods and analysis are appropriate for answering those questions. They also assess whether the paper answers such questions as:
- Is the study explained clearly enough and in enough detail that another researcher could replicate it?
- How does the study challenge or add to the body of knowledge on this topic?
- Does it fit the standards and scope of the journal to which it was submitted?
- If the study involves humans or animals, did the authors acquire the required approvals and meet ethical standards?
- Does it give proper attribution to earlier research?
When German theologian Henry Oldenburg created the first journal dedicated to science in 1665, he considered the key functions of a research journal to be registration, certification, dissemination and archiving, writes Robert Campbell, a senior publisher at Wiley-Blackwell Publishing, in the book Academic and Professional Publishing.
Peer review is considered the gold standard for assessing research content, Sullenberger explained in an email interview. But journalists must understand it is not infallible, she added.
“Science is self-correcting through replication and reproducibility, and research fraud can be difficult to detect in peer review,” she wrote.
2. Journalists can help the public recognize the value of peer review by noting whether the studies they cover have been peer reviewed.
Scholars, research organizations and others regularly criticize news outlets for failing to explain whether new research they report on or the older studies they incorporate into their stories have undergone peer review. It’s important that journalists differentiate between peer-reviewed research and preprint papers, which often present preliminary findings.
Sullenberger told JR: “Greater clarity when journalists cover unreviewed preprints is needed; they should not be reported as having the same validity and authority as peer-reviewed research papers. “
A recent study in the journal Health Communication finds that many of the news articles written about COVID-related preprints during the first four months of 2020 did not indicate the scientific uncertainty of that research. About 43% of the stories analyzed did not mention the research was a preprint, unreviewed, preliminary or in need of verification.
At the time of that study, however, many of the journalists drawn into reporting the frenzy of stories about the pandemic were unfamiliar with preprints, Inglis says. Today, he adds, journalists covering the coronavirus are much more likely to include phrases such as “not yet peer reviewed” to describe preprints.
Sense About Science urges the public to pay attention to whether a study being discussed in a government meeting or in the media has been peer reviewed. “The more we ask, ‘is it peer reviewed?’ the more obliged reporters will be to include this information,” the organization asserts in a leaflet it created to help the public scrutinize the scientific information featured in news stories.
Knowing whether research has been peer reviewed helps a person judge how much weight to give the claims being made by its authors, Tracey Brown, the managing director of Sense About Science, explained during an interview with The Scholarly Kitchen blog.
“We have to establish an understanding that the status of research findings is as important as the findings themselves,” Brown says in a prepared statement. “This understanding has the capacity to improve the decisions we make across all of society.”
3. Peer reviewers help decide a study’s fate.
Journal editors typically assign two or more reviewers to each research paper. Some also employ a statistical specialist.
While the selection process differs, journals choose reviewers based on factors such as expertise, reputation and the journal’s prior experience with the reviewer. While it can be difficult to recruit scientists willing to examine manuscripts because of the time required for proper scrutiny, many do it because of “a sense of duty to help advance their disciplines, as well as the need for reciprocity, knowing other researchers volunteer to peer review their manuscript submissions,” Science magazine reported earlier this year.
Reviewers can make recommendations about whether a journal should accept, reject or send a paper back for minor or major revisions. Reviewers usually submit reports offering their overall impressions of a paper and suggestions for improvements. Most often, though, the final decision lies with one or more of the journal’s editors or its editorial board.
Inglis, a former assistant editor of The Lancet who is now a publisher of five peer-reviewed journals, says a common criticism of the peer-review process is its lengthy timeline, which can span from weeks to a year or more. Another complaint: Sometimes, journals send a study back and notify the authors that they would be willing to accept or reconsider the paper for publication if the authors do more research.
“Sometimes, the demands made are completely unrealistic,” Inglis adds. “The criticism from the authors is that editors don’t know that when they say ‘Do this additional experiment,’ that’s another year [added to the timeline]. Meanwhile, the work is perfectly valid.”
Inglis says bioRxiv (pronounced “bio-archive”) and medRxiv (pronounced “med-archive”) were created so researchers could disseminate preliminary versions of their papers, allowing the scientific community to immediately use and start building on those findings and data.
4. The peer-review process varies significantly among academic journals.
There are several kinds of peer review, and journals often state on their websites which one they use. The most common are single-blinded peer review, which allows reviewers to know the authors’ identities while reviewers’ identities remain anonymous, and double-blinded peer review, in which authors and reviewers are unaware of each other’s identities.
Both have advantages. Advocates argue anonymity protects reviewers from retribution. It also helps shield authors from biases based on factors such as gender, nationality, language and affiliations with less prestigious institutions, Tony Ross-Hellauer, a postdoctoral researcher at the Know-Center in Austria, writes in “What is Open Peer Review? A Systematic Review,” published on the European open access platform F1000Research in 2017.
Keeping identities secret can create problems, however.
“At the editorial level, lack of transparency means that editors can unilaterally reject submissions or shape review outcomes by selecting reviewers based on their known preference for or aversion to certain theories and methods,” Ross-Hellauer writes. He adds that reviewers, “shielded by anonymity, may act unethically in their own interests by concealing conflicts of interest.”
A newer type of peer review, called open peer review, is not as prevalent. But the scientific community has ongoing discussions about whether its greater transparency might help improve research quality.
While there is no universally accepted definition of open peer review, also known as open identity peer review, the identities of both authors and reviewers typically are made known to each other. Ross-Hellauer notes that disclosing reviewers’ names may force them “to think more carefully about the scientific issues and to write more thoughtful reviews.”
A growing number of journals are posting not just the papers they accept but also the feedback peer reviewers gave the papers’ authors.
5. Peer review continues to evolve.
Some journals have started initiating peer review after a paper is published instead of beforehand, although this still is not common. MedEdPublish, an online scholarly journal, is one of those that employ post-publication peer review. Its papers undergo peer review on the website by members of the medical education community, which could include the journal’s editor, members of its editorial board or a panel of reviewers.
Under the MedEdPublish model, a paper has undergone formal peer review after at least two members of the journal’s review panel evaluate it. The paper can be critiqued and improved over time as a living document on the journal website.
“Post-publication peer review follows an open and transparent process, which aims to avoid editorial bias while increasing the speed of publication,” according to the website. “We use an ‘open identities’ principle, whereby all reviewers submit their feedback publicly, under their own name, and everyone visiting an article page can see all peer review reports, referee names, and comments, and can join the discussion if they wish.”
Another noteworthy shift: Some journals are working to diversify their pools of reviewers by ensuring women, racial and ethnic minorities, and scientists from other countries help appraise and select studies for publication.
Research indicates the overwhelming majority of experts chosen as reviewers are men. A study published earlier this year in Science Advances examines internal data for 145 scholarly journals across fields and finds that women comprised 21% of their reviewers between 2010 and 2016. At journals dedicated to biomedical and health research, 24.6% of reviewers were women.
The Lancet medical journal has set targets for increasing the number of women and scientists from low- and middle-income countries, Sabin, one of its senior editors, wrote in an email interview with JR. In 2019, The Lancet family of journals announced its Diversity Pledge.
“We track, monitor, and report representation of authors, reviewers, and editorial advisors by gender and across geography,” Sabin told JR in an e-mail.
She added that the journal formed a task force late last year to, among other things, examine its policies and processes to find ways to increase the representation of experts who are racial and ethnic minorities.
The Coalition for Diversity and Inclusion in Scholarly Communications has focused on the issue globally. More than 90 organizations have adopted the coalition’s Joint Statement of Principles, which aims to “promote involvement, innovation, and expanded access to leadership opportunities that maximize engagement across identity groups and professional levels.”
Identity groups include racial and sexual minorities, military veterans, pregnant women, parents and people from lower social classes and socioeconomic backgrounds.
The Journalist’s Resource would like to thank Rick Weiss, the director of SciLine, and Meredith Drosback, SciLine’s associate director of science, for their help in creating this tip sheet.
Expert Commentary