Mike Bisaccio Academic Publishing, Peer Review

PRW 2023: “Peer Review and The Future of Publishing”

Each year, our team at Cabells celebrates Peer Review Week (PRW) and recognizes the fact that so much of the work we do each day revolves around peer review, which is the backbone of scholarly communication and the key to maintaining research quality. The theme this year for PRW is “Peer Review and the Future of Publishing,” which would be an appropriate theme every year. To work as intended and as needed, peer review will need to continuously adapt and evolve along with publishing.

The importance of peer review to the quality and overall success of a journal can’t be overstated. For a journal to be recognized in the academic or medical community as legitimate, a robust peer review system must be in place. In recent years, the scholarly community has been shown time and again the results of substandard (or nonexistent) peer review. It has also become clear that identifying an effective and efficient model of peer review has proven to be a challenge for publishers.

Our friend, Daley White, a research scientific editor with the Moffitt Cancer Center, has written an excellent piece discussing the current state of peer review and highlighting a few promising alternative strategies. That piece, along with another by Daley discussing the role of generative artificial intelligence in peer review, should not be missed.

The bedrock of scholarly publishing

At its core, peer review is about benefiting the knowledge base by establishing quality control with respect to published research, which is then used to generate more knowledge. By publishing research papers that have been thoughtfully peer reviewed, academic journals make it possible for researchers around the world to learn about the latest findings in their field. This helps advance knowledge and to foster collaboration amongst researchers. Researchers, funders, and the public all expect that research has been reviewed, is sound, and worthy of being built upon.

Peer review helps to ensure published work is high-quality with findings that are accurate and reliable by helping to identify and correct errors, omissions, and biases. Ultimately, authors are responsible for conducting sound research and not fabricating data or results. Unfortunately, the immense pressure to publish along with the industry’s unwillingness to publish null results, both contribute to making this responsibility an insurmountable challenge for some.

For a journal to be considered for inclusion in Journaltyics, our evaluators must have evidence of a rigorous peer review system.

To be effective, peer review must be unbiased and transparent though the extent to which journals are open about their review process varies. Promoting and expanding transparency and accountability in the research and peer review processes shows readers how the paper was evaluated and helps them understand the reasons for its acceptance or rejection, which helps to build trust in the publication process and the research itself.

Time after time

Can it be assumed that peer review is consistently conducted with the necessary rigor when in most cases it is added to the workload of already very busy and time-strapped reviewers? Most workplaces don’t provide an allowance of time for peer review, and there is no compensation for conducting reviews. So, without incentives, peer review is conducted solely to contribute to a knowledge base that needs to be carefully managed and safeguarded.

Along with pressure on scholars to find the time to conduct reviews, there is pressure on journals to review papers quickly. But can speed be reconciled with quality? Speedy peer review, when taken to an extreme, is an indication of the type of substandard or virtually nonexistent peer review often found in predatory journals.

While it’s important to authors that articles are published in a timely manner (which requires timely peer review), there is a correlation between speed and quality that the industry as a whole is working under. Often, the state of a journals peer review process comes down to which journals have more resources available. Not all journals can swing having an in-house statistician to review research statistics on staff. Training in peer review as part of PhD programs would also be valuable – while early career researchers are very knowledgeable in their fields despite being relatively inexperienced, having ECR’s conduct peer review with no training is less than optimal.

So, this PRW we will consider these and other ideas as we continue our work as champions of peer review – and Cabells team member Clarice continues her work as a member of the PRW Steering Committee. Our work at Cabells will adapt and evolve right along with peer review and publishing into the future. What won’t change is the key role played by peer review in maintaining the quality, transparency, and accountability of research and the integrity of knowledge.

daleywhite Peer Review

Innovations in Peer Review: Increasing Capacity While Improving Quality

Peer review is a critical aspect of modern academic research, but it’s no secret that journals are struggling to provide high-quality and timely peer review for submitted manuscripts. It’s clear that changes are needed to increase the capacity and efficiency of peer review without reducing the quality of the review. However, several alternative peer review models are up to the challenge. We’ll discuss the most well-established alternative peer review strategies, identify some commonalities between models, and provide key takeaways for everyone in academia.

The Current State of Peer Review

Before we can discuss new innovations, it’s important to evaluate the modern peer review structure. Peer review serves as a vetting process for journals to filter out research manuscripts that are considered unsuitable for their readership, whether that’s because of poorly defined methods, suspicious or fraudulent results, a lack of supporting evidence or proof, or unconstructive findings. After peer reviewers read and provide their criticism of a manuscript, they’ll generally advise journals to 1) accept a submission as-is, 2) accept a submission with minor revisions, 3) request major revisions before reevaluating the suitability of a paper for publication, or 4) outright reject a manuscript. Manuscripts will often go through two or three rounds of peer review, usually with the same peer reviewers, before a paper is ready for publication.

Most medical journals require at least two experts in a related field to review a manuscript. This is typically done through anonymized peer review, in which the authors don’t know the identity of the reviewers, but the open peer review model (in which the identity of peer reviewers is known to the author, with or without their reviews being publicly available following manuscript publication) has been gaining traction in recent years.

The Problems with Modern Peer Review

As the academic publishing industry rapidly expands and becomes increasingly digital, the current peer review model has been struggling to keep up. Peer review is resource-intensive, especially in money and time. Peer review is voluntary and reviewers are almost universally not compensated for their contributions, leading to a lack of motivation to participate, especially given the time and effort peer review requires. The lack of transparency in peer review has also been increasingly criticized in recent years because it can lead to biased reviews and unequal standards. Despite this, many academicians place unwarranted trust in the validity and efficacy of the peer review process.

On top of all of this, the peer review process is notoriously slow. This is usually attributed to the shortage of qualified peer reviewers, and with good reason: a 2016 survey found that 20% of individuals performed 69% to 94% of reviews. It’s a tough problem to tackle, but there are some innovative new peer review strategies that aim to improve the timeliness and accessibility of peer review, maximize the effective use of peer reviewers’ time, and maintain or improve upon modern quality expectations.

Alternative Strategies for Peer Review

Portable peer review

Overview: Authors pay a company to perform independent, single-blind (ie, anonymized), unbiased peer review, which is then shared with journals at the time of submission. A subset of this is Peer Review by Endorsement (also called Author-Guided Peer Review), in which authors request their peers to review their manuscripts, which are then provided to journals.

Pros: Journals aren’t responsible for coordinating peer reviews; avoids redundancy of multiple sets of peer reviewers evaluating the same paper for different journals

Cons: Additional fee is burdensome for authors, especially as article processing charges become more common; not many journals currently accept externally provided peer reviews; potential bias for Peer Review by Endorsement

Pre–peer review commenting

Overview: Informal community input is given on a manuscript while authors are simultaneously submitting the paper to journals. This input can be either open (eg, publicly available materials for anyone to comment on) or closed (eg, materials are shown only to a select group of commenters). You may be familiar with a common pre–peer review commenting platform without even knowing it: preprints, and many of the same pros and cons apply here.

Pros: Strengthens the quality of a paper before journal evaluation; faster than traditional peer review; typically involves low costs; may include moderators who filter out unconstructive comments

Cons: Allows non-experts to voice incorrect opinions; reduces editorial control; introduces threat of plagiarism or scooping; may make faulty or inadequate science publicly available

Post-publication commenting

Overview: Peer review takes place after the manuscript has already been published by a journal. Editors invite a group of qualified experts in the field to provide feedback on the publication. Manuscripts may or may not receive some level of peer review before publication.

Pros: Reduces time delay for peer review; comments are typically public and transparent; theoretically provides continuous peer review as new developments and discoveries are made, which may support, disprove, change, or otherwise affect research findings

Cons: Requires buy-in from many peer reviewers who are willing to review; faulty or inadequate science may be made publicly available; can become resource-intensive, especially time-intensive

Registered Reports

Overview: Studies are registered with a journal before research is performed and undergo peer review both before and after research is conducted. The first round of peer review focuses on the quality of the methods, hypothesis, and background, and the second round focuses on the findings.

Pros: Papers are typically guaranteed acceptance with the journal; each round of peer review hypothetically requires less time/effort; papers are typically more thorough and scientifically sound; provides research support and limited mentorship, which can be especially valuable for early-career investigators

Cons: Two rounds of peer review are required instead of one; reduces procedural flexibility; logistical delays are common; seen as inefficient for sequential or ongoing research.

Artificial Intelligence–Assisted Review

Overview: Artificial intelligence and machine learning software are developed to catch common errors or shortcomings, allowing peer reviewers to focus on more conceptually-based criticism, such as the paper’s novelty, rigor, and potential impact. This strategy is more widely seen in humanities and social sciences research.

Pros: Increases efficient use of peer reviewers’ time; improves standardization of review; can automate processes like copyediting or formatting

Cons: Requires extensive upfront cost and development time as well as ongoing maintenance; prone to unintentional bias; ethically dubious; requires human oversight

Commonalities and Takeaways

There are a few key similarities and fundamental practices that are found throughout several of the peer review strategies discussed above:

  1. Peer reviewer compensation, either in the form of financial compensation or public recognition/resume material— though this can often cause its own problems
  2. Decoupling the peer review process from the publication process
  3. Expanding the diversity of peer reviewers
  4. Improving transparency of peer review
  5. Improving standardization of peer review, often through paper priority scores or weighted reviewer scoring based on review evaluation ratings/reputation

The key overall takeaway from these new strategies? Change may be slow, but it’s certainly coming. More and more journals are embracing shifts in peer review, such as the growing traction of transferable peer review (i.e., if a manuscript is transferred between journals, any available reviewers’ comments will be shared with the new journal) and the transition from anonymous to open identification of reviewers, and most experts agree that peer review practices will continue to change in upcoming years.

If you’re interested in becoming more involved in leading the evolution of peer review, take some time to research the many proposed alternative peer review strategies. Try to start conversations about new peer review models in academic spaces to spread the word about alternative strategies. If you’re able, try to participate in validated, evidence-driven research to either validate the efficacy of alternative peer review models or demonstrate the inefficiency of our current structure. Change always requires motivated and driven individuals who are willing to champion the cause. The communal push toward revolutionizing peer review is clearly growing—now, it’s up to the community to determine which model will prevail.

daleywhite Peer Review

The Role of Generative Artificial Intelligence in Peer Review

This year, as the research community’s trust in the peer review system’s efficacy and efficiency has wavered, we’ve seen a sharp rise in the proposal and implementation of alterations to the standard peer review process. As such, it’s not surprising that the community-selected theme for the 2023 Peer Review Week is “Peer Review and The Future of Publishing.” When taken in context with the runner-up topics— “Peer Review and Technology” and “Ethical Issues in Peer Review”—it’s clear that the medical community is uncertain about many of these changes, especially changes that involve new and unproven technology. In this article, we’ll narrow our focus to a specific topic that embodies much of the potential (both positive and negative) of these changes: the role of generative artificial intelligence (AI) in peer review.

Artificial Intelligence in Peer Review

Generative AI’s potential role in peer review is complex, with the capacity for great time-saving efficiency as well as for severe ethical violations and misinformation. In theory, generative AI platforms could be used throughout the peer review process, from the initial drafting to the finalization of a decision letter or a reviewer’s critiques. An editor or reviewer could input a manuscript (either in whole or individual sections) into a generative AI platform and then prompt the tool for either an overall review of the paper or for a specific analysis, such as evaluating the reproducibility of the article’s methods or the language clarity. However, this comes with a myriad of potential benefits and drawbacks.

Arguments in support of generative AI in peer review include:

  • Automation of time-intensive tasks, thereby reducing the extensive turnaround windows for manuscript evaluation
  • The rich potential of AI as a supportive tool, not as a total replacement for editors or reviewers
  • Use of AI to draft and refine decision letters and reviewer comments

Conversely, arguments in opposition to generative AI in peer review include:

  • Potential for unreliable, factually incorrect output
  • Discrimination resulting from large language models’ tendency toward biases
  • Non-confidentiality of valuable research data and proprietary information
  • Murky status of autogenerated content as plagiarism

Current State of Generative AI in Peer Review

The debate on whether generative AI should be permissible for peer review has waged for most of 2023, and in recent months, key funders have announced their stance. Foremost among them is the National Institutes of Health (NIH), the largest funder of medical research in the world. In June of 2023, the NIH banned the use of generative AI during peer review, citing confidentiality and security as primary concerns; a Security, Confidentiality and Nondisclosure Agreement stipulating that AI tools are prohibited was then sent all NIH peer reviewers. The Australian Research Council followed quickly afterwards with a similar ban. Other funding bodies, such as the United States’ National Science Foundation and the European Research Council, currently have working groups developing position statements regarding generative AI use for peer review.

Publishers, however, are placed in a unique position. Some journals have proposed adopting generative AI tools to augment the current peer review process and to automate some processes that are currently completed by editors or reviewers, which could meaningfully shorten the time required to complete a thorough peer review. Currently, few publishers have posted public position statements regarding the use of generative AI during peer review; an exception is Elsevier, who has stated that book and commissioned content reviewers are not permitted to use generative AI due to confidentiality concerns. The future of generative AI integration into journals’ manuscript evaluation workflows remains unclear.

Understanding the 2023 Theme Beyond Generative AI

Beyond the proposed role of generative AI and natural language processing in peer review, the 2023 theme of “Peer Review and The Future of Publishing” encompasses a wide range of current and anticipated shifts in the publishing process. These changes can have a domino effect to sway the community’s opinion on generative AI, potentially swinging the needle regarding its use during peer review. Other related considerations include:

Each of these trends will affect peer review in crucial but unclear ways, which has led to a heightened sense of uncertainty regarding peer review throughout the medical research community. The 2023 theme for Peer Review Week aims to hold space for these concerns and allow stakeholders to collaboratively discuss the most effective routes forward to ensure that peer review is an effective and efficient process.