Mike Bisaccio Academic Publishing, Peer Review

PRW 2023: “Peer Review and The Future of Publishing”

Each year, our team at Cabells celebrates Peer Review Week (PRW) and recognizes the fact that so much of the work we do each day revolves around peer review, which is the backbone of scholarly communication and the key to maintaining research quality. The theme this year for PRW is “Peer Review and the Future of Publishing,” which would be an appropriate theme every year. To work as intended and as needed, peer review will need to continuously adapt and evolve along with publishing.

The importance of peer review to the quality and overall success of a journal can’t be overstated. For a journal to be recognized in the academic or medical community as legitimate, a robust peer review system must be in place. In recent years, the scholarly community has been shown time and again the results of substandard (or nonexistent) peer review. It has also become clear that identifying an effective and efficient model of peer review has proven to be a challenge for publishers.

Our friend, Daley White, a research scientific editor with the Moffitt Cancer Center, has written an excellent piece discussing the current state of peer review and highlighting a few promising alternative strategies. That piece, along with another by Daley discussing the role of generative artificial intelligence in peer review, should not be missed.

The bedrock of scholarly publishing

At its core, peer review is about benefiting the knowledge base by establishing quality control with respect to published research, which is then used to generate more knowledge. By publishing research papers that have been thoughtfully peer reviewed, academic journals make it possible for researchers around the world to learn about the latest findings in their field. This helps advance knowledge and to foster collaboration amongst researchers. Researchers, funders, and the public all expect that research has been reviewed, is sound, and worthy of being built upon.

Peer review helps to ensure published work is high-quality with findings that are accurate and reliable by helping to identify and correct errors, omissions, and biases. Ultimately, authors are responsible for conducting sound research and not fabricating data or results. Unfortunately, the immense pressure to publish along with the industry’s unwillingness to publish null results, both contribute to making this responsibility an insurmountable challenge for some.

For a journal to be considered for inclusion in Journaltyics, our evaluators must have evidence of a rigorous peer review system.

To be effective, peer review must be unbiased and transparent though the extent to which journals are open about their review process varies. Promoting and expanding transparency and accountability in the research and peer review processes shows readers how the paper was evaluated and helps them understand the reasons for its acceptance or rejection, which helps to build trust in the publication process and the research itself.

Time after time

Can it be assumed that peer review is consistently conducted with the necessary rigor when in most cases it is added to the workload of already very busy and time-strapped reviewers? Most workplaces don’t provide an allowance of time for peer review, and there is no compensation for conducting reviews. So, without incentives, peer review is conducted solely to contribute to a knowledge base that needs to be carefully managed and safeguarded.

Along with pressure on scholars to find the time to conduct reviews, there is pressure on journals to review papers quickly. But can speed be reconciled with quality? Speedy peer review, when taken to an extreme, is an indication of the type of substandard or virtually nonexistent peer review often found in predatory journals.

While it’s important to authors that articles are published in a timely manner (which requires timely peer review), there is a correlation between speed and quality that the industry as a whole is working under. Often, the state of a journals peer review process comes down to which journals have more resources available. Not all journals can swing having an in-house statistician to review research statistics on staff. Training in peer review as part of PhD programs would also be valuable – while early career researchers are very knowledgeable in their fields despite being relatively inexperienced, having ECR’s conduct peer review with no training is less than optimal.

So, this PRW we will consider these and other ideas as we continue our work as champions of peer review – and Cabells team member Clarice continues her work as a member of the PRW Steering Committee. Our work at Cabells will adapt and evolve right along with peer review and publishing into the future. What won’t change is the key role played by peer review in maintaining the quality, transparency, and accountability of research and the integrity of knowledge.

daleywhite Peer Review

Innovations in Peer Review: Increasing Capacity While Improving Quality

Peer review is a critical aspect of modern academic research, but it’s no secret that journals are struggling to provide high-quality and timely peer review for submitted manuscripts. It’s clear that changes are needed to increase the capacity and efficiency of peer review without reducing the quality of the review. However, several alternative peer review models are up to the challenge. We’ll discuss the most well-established alternative peer review strategies, identify some commonalities between models, and provide key takeaways for everyone in academia.

The Current State of Peer Review

Before we can discuss new innovations, it’s important to evaluate the modern peer review structure. Peer review serves as a vetting process for journals to filter out research manuscripts that are considered unsuitable for their readership, whether that’s because of poorly defined methods, suspicious or fraudulent results, a lack of supporting evidence or proof, or unconstructive findings. After peer reviewers read and provide their criticism of a manuscript, they’ll generally advise journals to 1) accept a submission as-is, 2) accept a submission with minor revisions, 3) request major revisions before reevaluating the suitability of a paper for publication, or 4) outright reject a manuscript. Manuscripts will often go through two or three rounds of peer review, usually with the same peer reviewers, before a paper is ready for publication.

Most medical journals require at least two experts in a related field to review a manuscript. This is typically done through anonymized peer review, in which the authors don’t know the identity of the reviewers, but the open peer review model (in which the identity of peer reviewers is known to the author, with or without their reviews being publicly available following manuscript publication) has been gaining traction in recent years.

The Problems with Modern Peer Review

As the academic publishing industry rapidly expands and becomes increasingly digital, the current peer review model has been struggling to keep up. Peer review is resource-intensive, especially in money and time. Peer review is voluntary and reviewers are almost universally not compensated for their contributions, leading to a lack of motivation to participate, especially given the time and effort peer review requires. The lack of transparency in peer review has also been increasingly criticized in recent years because it can lead to biased reviews and unequal standards. Despite this, many academicians place unwarranted trust in the validity and efficacy of the peer review process.

On top of all of this, the peer review process is notoriously slow. This is usually attributed to the shortage of qualified peer reviewers, and with good reason: a 2016 survey found that 20% of individuals performed 69% to 94% of reviews. It’s a tough problem to tackle, but there are some innovative new peer review strategies that aim to improve the timeliness and accessibility of peer review, maximize the effective use of peer reviewers’ time, and maintain or improve upon modern quality expectations.

Alternative Strategies for Peer Review

Portable peer review

Overview: Authors pay a company to perform independent, single-blind (ie, anonymized), unbiased peer review, which is then shared with journals at the time of submission. A subset of this is Peer Review by Endorsement (also called Author-Guided Peer Review), in which authors request their peers to review their manuscripts, which are then provided to journals.

Pros: Journals aren’t responsible for coordinating peer reviews; avoids redundancy of multiple sets of peer reviewers evaluating the same paper for different journals

Cons: Additional fee is burdensome for authors, especially as article processing charges become more common; not many journals currently accept externally provided peer reviews; potential bias for Peer Review by Endorsement

Pre–peer review commenting

Overview: Informal community input is given on a manuscript while authors are simultaneously submitting the paper to journals. This input can be either open (eg, publicly available materials for anyone to comment on) or closed (eg, materials are shown only to a select group of commenters). You may be familiar with a common pre–peer review commenting platform without even knowing it: preprints, and many of the same pros and cons apply here.

Pros: Strengthens the quality of a paper before journal evaluation; faster than traditional peer review; typically involves low costs; may include moderators who filter out unconstructive comments

Cons: Allows non-experts to voice incorrect opinions; reduces editorial control; introduces threat of plagiarism or scooping; may make faulty or inadequate science publicly available

Post-publication commenting

Overview: Peer review takes place after the manuscript has already been published by a journal. Editors invite a group of qualified experts in the field to provide feedback on the publication. Manuscripts may or may not receive some level of peer review before publication.

Pros: Reduces time delay for peer review; comments are typically public and transparent; theoretically provides continuous peer review as new developments and discoveries are made, which may support, disprove, change, or otherwise affect research findings

Cons: Requires buy-in from many peer reviewers who are willing to review; faulty or inadequate science may be made publicly available; can become resource-intensive, especially time-intensive

Registered Reports

Overview: Studies are registered with a journal before research is performed and undergo peer review both before and after research is conducted. The first round of peer review focuses on the quality of the methods, hypothesis, and background, and the second round focuses on the findings.

Pros: Papers are typically guaranteed acceptance with the journal; each round of peer review hypothetically requires less time/effort; papers are typically more thorough and scientifically sound; provides research support and limited mentorship, which can be especially valuable for early-career investigators

Cons: Two rounds of peer review are required instead of one; reduces procedural flexibility; logistical delays are common; seen as inefficient for sequential or ongoing research.

Artificial Intelligence–Assisted Review

Overview: Artificial intelligence and machine learning software are developed to catch common errors or shortcomings, allowing peer reviewers to focus on more conceptually-based criticism, such as the paper’s novelty, rigor, and potential impact. This strategy is more widely seen in humanities and social sciences research.

Pros: Increases efficient use of peer reviewers’ time; improves standardization of review; can automate processes like copyediting or formatting

Cons: Requires extensive upfront cost and development time as well as ongoing maintenance; prone to unintentional bias; ethically dubious; requires human oversight

Commonalities and Takeaways

There are a few key similarities and fundamental practices that are found throughout several of the peer review strategies discussed above:

  1. Peer reviewer compensation, either in the form of financial compensation or public recognition/resume material— though this can often cause its own problems
  2. Decoupling the peer review process from the publication process
  3. Expanding the diversity of peer reviewers
  4. Improving transparency of peer review
  5. Improving standardization of peer review, often through paper priority scores or weighted reviewer scoring based on review evaluation ratings/reputation

The key overall takeaway from these new strategies? Change may be slow, but it’s certainly coming. More and more journals are embracing shifts in peer review, such as the growing traction of transferable peer review (i.e., if a manuscript is transferred between journals, any available reviewers’ comments will be shared with the new journal) and the transition from anonymous to open identification of reviewers, and most experts agree that peer review practices will continue to change in upcoming years.

If you’re interested in becoming more involved in leading the evolution of peer review, take some time to research the many proposed alternative peer review strategies. Try to start conversations about new peer review models in academic spaces to spread the word about alternative strategies. If you’re able, try to participate in validated, evidence-driven research to either validate the efficacy of alternative peer review models or demonstrate the inefficiency of our current structure. Change always requires motivated and driven individuals who are willing to champion the cause. The communal push toward revolutionizing peer review is clearly growing—now, it’s up to the community to determine which model will prevail.

daleywhite Peer Review

The Role of Generative Artificial Intelligence in Peer Review

This year, as the research community’s trust in the peer review system’s efficacy and efficiency has wavered, we’ve seen a sharp rise in the proposal and implementation of alterations to the standard peer review process. As such, it’s not surprising that the community-selected theme for the 2023 Peer Review Week is “Peer Review and The Future of Publishing.” When taken in context with the runner-up topics— “Peer Review and Technology” and “Ethical Issues in Peer Review”—it’s clear that the medical community is uncertain about many of these changes, especially changes that involve new and unproven technology. In this article, we’ll narrow our focus to a specific topic that embodies much of the potential (both positive and negative) of these changes: the role of generative artificial intelligence (AI) in peer review.

Artificial Intelligence in Peer Review

Generative AI’s potential role in peer review is complex, with the capacity for great time-saving efficiency as well as for severe ethical violations and misinformation. In theory, generative AI platforms could be used throughout the peer review process, from the initial drafting to the finalization of a decision letter or a reviewer’s critiques. An editor or reviewer could input a manuscript (either in whole or individual sections) into a generative AI platform and then prompt the tool for either an overall review of the paper or for a specific analysis, such as evaluating the reproducibility of the article’s methods or the language clarity. However, this comes with a myriad of potential benefits and drawbacks.

Arguments in support of generative AI in peer review include:

  • Automation of time-intensive tasks, thereby reducing the extensive turnaround windows for manuscript evaluation
  • The rich potential of AI as a supportive tool, not as a total replacement for editors or reviewers
  • Use of AI to draft and refine decision letters and reviewer comments

Conversely, arguments in opposition to generative AI in peer review include:

  • Potential for unreliable, factually incorrect output
  • Discrimination resulting from large language models’ tendency toward biases
  • Non-confidentiality of valuable research data and proprietary information
  • Murky status of autogenerated content as plagiarism

Current State of Generative AI in Peer Review

The debate on whether generative AI should be permissible for peer review has waged for most of 2023, and in recent months, key funders have announced their stance. Foremost among them is the National Institutes of Health (NIH), the largest funder of medical research in the world. In June of 2023, the NIH banned the use of generative AI during peer review, citing confidentiality and security as primary concerns; a Security, Confidentiality and Nondisclosure Agreement stipulating that AI tools are prohibited was then sent all NIH peer reviewers. The Australian Research Council followed quickly afterwards with a similar ban. Other funding bodies, such as the United States’ National Science Foundation and the European Research Council, currently have working groups developing position statements regarding generative AI use for peer review.

Publishers, however, are placed in a unique position. Some journals have proposed adopting generative AI tools to augment the current peer review process and to automate some processes that are currently completed by editors or reviewers, which could meaningfully shorten the time required to complete a thorough peer review. Currently, few publishers have posted public position statements regarding the use of generative AI during peer review; an exception is Elsevier, who has stated that book and commissioned content reviewers are not permitted to use generative AI due to confidentiality concerns. The future of generative AI integration into journals’ manuscript evaluation workflows remains unclear.

Understanding the 2023 Theme Beyond Generative AI

Beyond the proposed role of generative AI and natural language processing in peer review, the 2023 theme of “Peer Review and The Future of Publishing” encompasses a wide range of current and anticipated shifts in the publishing process. These changes can have a domino effect to sway the community’s opinion on generative AI, potentially swinging the needle regarding its use during peer review. Other related considerations include:

Each of these trends will affect peer review in crucial but unclear ways, which has led to a heightened sense of uncertainty regarding peer review throughout the medical research community. The 2023 theme for Peer Review Week aims to hold space for these concerns and allow stakeholders to collaboratively discuss the most effective routes forward to ensure that peer review is an effective and efficient process.

Simon Linacre Academic Publishing, Peer Review, Scholarly Communication

Peer Review Week 2021: Identity in Peer Review

Peer Review Week 2021 has been announced for September 20–24 with the theme of Identity in Peer Review. Simon Linacre, who volunteers for the event’s Steering Committee, takes a look at the importance of the event and this year’s chosen theme.

For those new to scholarly communication, the annual celebration of peer review probably seems one of the more unlikely events to occur in the crowded calendar. It makes sense for relatively novel ideas such as open access and open science to have their day – or week – in the sun in October, while other events supporting academic research and universities in general pepper the rest of the year. So why is boring old peer review so special?

Well, it may be a surprise to learn it’s not that old, and when you dig deeper you find it is anything but boring. While journals began life in the 17th Century – 1665, to be precise – it seems the first peer reviews only took place in the 18th Century, and external reviews in the Victorian period. According to academic publishing historian Alex Csiszar, peer reviews grew from these beginnings very slowly, and only took hold in mainstream science journals in the post-war period.

Furthermore, this year’s theme shows that issues and challenges facing the world today are very much relevant to the process of peer review. Identity in Peer Review was the first Peer Review Week theme to be chosen by the public, and will explore the role of both personal and social identity in peer review. It is hoped that the various events and activities during the week will develop a more diverse, equitable and inclusive approach to peer review. Academia has seen increased emphasis on the taking of steps to ensure research literature reflects and amplifies diverse voices, and of course the manner in which peer review is conducted is key to that.

Peer Review Week steering committee co-chair Danielle Padula says: “If the past year has taught us anything, I think it’s that recognizing the composite of identities that make up who we are as individuals, organizations, and populations, and the links between those identities, is essential to the future of scholarship and, ultimately, global progress. The pandemic has illuminated myriad deep-seated inequities that we need to address in all areas of society, with academia being no exception. And I think that starts with unpacking various aspects of personal and social identity and how we need to rethink the systems in which we operate to acknowledge and make space for diverse identities.”

Looking back to learn about the future is an apt approach, given that the past of peer review is not far behind us, and radical change potentially so near in the future. As ever, focusing on peer review makes a lot of sense for everyone with an interest in knowledge sharing and scholarly communications. Roll on September.

If you are interested in learning more or volunteering, please visit the Peer Review Week website, or you can contact Danielle Padula (dpadula@scholasticahq.com) or Jayashree Rajagopalan (jayashreer@cactusglobal.com), who are co-chairing this year’s PRW steering committee.

Simon Linacre Academic Publishing, Peer Review, Scholarly Communication

How do you know you can trust a journal?

As many readers know, this week is Peer Review Week, the annual opportunity for those involved in scholarly communication and research to celebrate and learn about all aspects of peer review. As part of this conversation, Simon Linacre reflects on this year’s theme of ‘Trust in Peer Review’ in terms of the important role of peer review in the validation of scholarship, and dangers of predatory behaviour in its absence.

I was asked to deliver a webinar recently to a community of scholars in Eastern Europe and, as always with webinars, I was very worried about the Q&A section at the end. When you deliver a talk in person, you can tell by looking at the crowd what is likely to happen at the end of the presentation and can prepare yourself. A quiet group of people means you may have to ask yourself some pretty tough questions, as no one will put their hand up at the end to ask you anything; a rowdy crowd is likely to throw anything and everything at you. With a webinar, there are no cues, and as such, it can be particularly nerve-shredding.

With the webinar in question, I waited a while for a question and was starting to prepare my quiet crowd response, when a single question popped up in the chat box:

How do you know you can trust a journal?

As with all the best questions, this floored me for a while. How do you know? The usual things flashed across my mind: reputation, whether it’s published known scholars in its field, whether it is indexed by Cabells or other databases, etc. But suddenly the word trust felt a lot more personal than simply a tick box exercise to confirm a journal’s standing. That may confirm it is trustworthy but is that the same as the feeling an individual has when they really trust something or someone?

The issue of trust is often the unsaid part of the global debates that are raging currently, whether it is responses to the coronavirus epidemic, climate change or democracy. Politicians, as always, want the people to trust them; but increasingly their actions seem to be making that trust harder and harder. As I write, the UK put its two top scientists in front of the cameras to give a grave warning about COVID-19 and a second wave of cases. The fact there was no senior politician to join them was highly symbolic.

It is with this background that the choice of the theme Trust in Peer Review is an appropriate one for Peer Review Week (full disclosure: I have recently joined one of the PRW committees to support the initiative). There is a huge groundswell of support by publishers, editors and academics to support both the effectiveness of peer review and the unsung heroes who do the job for little recognition or reward. The absence of which would have profound implications for research and society as a whole.

Which brings me to the answer to the question posed above, which is to ask the opposite: how do you know when you cannot trust a journal? This is easier to answer as you can point to all those characteristics and behaviours that you would want in a journal. We see on a daily basis with our work on Predatory Reports how the absence of crucial aspects of a journal’s workings can cause huge problems for authors. No listed editor, a fake editorial board, a borrowed ISSN, a hijacked journal identity, a made-up impact factor, and – above all – false promises of a robust peer review process. Trust in peer review may require some research on the part of the author in terms of checking the background of the journal, its publisher and its editors, and it may require you to contact the editor, editorial board members or published authors to get personal advice on publishing in that journal. But doing that work in the first place and receiving personal recommendations will build trust in peer review for any authors who have doubts – and collectively for all members of the academic community.

Simon Linacre Peer Review

Still without peer?

Next week the annual celebration of peer review takes place, which despite being centuries old is still an integral part of scholarly communications. To show Cabells’ support of #PeerReviewWeek, Simon Linacre looks at why peer review deserves its week in the calendar and to survive for many years to come.

I was recently asked by Cabells’ partners Editage to upload a video to YouTube explaining how the general public benefited from peer review. This is a good question, because I very much doubt the general public is aware at all of what peer review is and how it impacts their day-to-day lives. But if you reflect for just a moment, it is clear it impacts almost everything, much of which is taken for granted on a day-to-day basis.

Take making a trip to the shops. A car is the result of thousands of experiments and validated peer review research over a century to come up with the safest and most efficient means of driving people and things from one place to another; each supermarket product has been health and safety tested; each purchase uses digital technology such as the barcode that has advanced through the years to enable fast and accurate purchasing; even the license plate recognition software that gives us a ticket when we stay too long in the car park will be a result of some peer reviewed research (although most people may struggle to describe that as a ‘benefit’).

So, we do all benefit from peer review, even if we do not appreciate it all the time. Does that prove the value of peer review? For some, it is still an inefficient system for scholarly communications, and over the years a number of platforms have sought to disrupt it. For example, PLoS has been hugely successful as a publishing platform where a ‘light touch peer review’ has taken place to enable large-scale, quick turnaround publishing. More recently, F1000 has developed a post-publication peer review platform where all reviews are visible and take place on almost all articles that are submitted. While these platforms have undoubtedly offered variety and author choice to scientific publishing processes, they have yet to change the game, particularly in social sciences where more in-depth peer review is required.

Perhaps real disruption will be seen to accommodate peer review rather than change it. This week’s announcement at the ALPSP Conference by Cactus Communications – part of the same organization as Editage – of an AI-powered platform that can allow authors to submit articles to be viewed by multiple journal editors may just change the way peer review works. Instead of the multiple submit-review-reject cycles authors have to endure, they can submit their article to a system that can check for hygiene factor quality characteristics and relevance to journals’ coverage, and match them with potentially interested editors who can offer the opportunity for the article to then be peer reviewed.

If it works across a good number of journals, one can see that from the perspective of authors, editors and publishers, it would be a much more satisfactory process than the traditional one that still endures. And a much quicker one to boot, which means that the general public should see the benefits of peer review all the more speedily.