Unveiling Perspectives on Peer Review and Research Integrity: Survey Insights – The Scholarly Kitchen – Inergency

I recently co-authored a paper published in Learned Publishing, “Adapting Peer Review for the Future: Digital Disruptions and Trust in Peer Review”, which addresses the intersection of digital disruptions and trust in peer review. The ongoing discourse on the integration of artificial intelligence (AI) in academic publishing has generated diverse opinions, with individuals expressing both support and opposition to its adoption.

The scrutiny of peer review and research integrity has raised questions both in the presence and absence of AI. Is the current inquiry into research integrity during peer review solely prompted by the advent of AI, or has it always been a concern, considering past incidents involving retractions, plagiarism, and other unethical practices?

Harini Calamur, the co-author of the paper says:

“The research integrity problem has several facets. At the top is the sheer volume of papers to be peer reviewed, and the lack of enough peer reviewers. It is next to impossible to be true to each and every paper, and to ensure that everything is as per requirement. Unless we fix that, we are not going to know which paper has dodgy ethics, and which does not. And therefore, institutional science will always be playing a catch-up game in terms of research integrity. The integration of AI offers potential solutions, but it’s important to recognize that technological advancements alone cannot fully address these structural challenges. Fundamental changes in the peer review process are essential for a proactive stance on research integrity.”

Indeed, research integrity has consistently been a cause of concern. Therefore, the blame lies not with AI but rather with the established processes, particularly as the dynamics of academia undergo transformations, pushing against traditional boundaries and introducing a multitude of new opportunities and challenges.

In the era of digital advancements and the expansive information landscape, it becomes imperative to establish robust mechanisms that uphold research integrity and ensure the trustworthiness of academic output. While this is not a novel concern, adapting to this evolving landscape requires time and resources. Take Google, for example. Despite being a valuable information resource, the sheer volume of data makes it challenging to discern which information it offers via search results is credible. The realm of research increasingly resembles this scenario, with continually growing numbers of academic papers published annually. This expansion of literature has strained the peer review process, burdening experienced reviewers and leading to either hasty evaluations or delays in publication. Addressing the implications of the ever-growing volume of publication is essential for maintaining the integrity of academic publishing. Hence, the question arises: How can we refocus the community on research integrity?

To gain a deeper understanding of the challenges and opportunities within the evolving academic landscape, Cactus Communications (CACTUS – full disclosure, my employer) conducted a comprehensive global survey during Peer Review Week 2022. The survey, encompassing 852 participants from diverse academic backgrounds and career stages, explored their perspectives on research integrity and the efficacy of the peer review process.

Figure 1 is a visual representation of participant opinions on the issues that peer review is expected to address. Notably, 78.83% of participants expressed the view that peer review should include identifying plagiarism. But does this really fall in the purview of what a peer reviewer is expected to do? These findings set the stage for a more thorough examination of the expectations and challenges associated with upholding research integrity through the peer review process. What is the responsibility of the peer reviewers and what should instead be handled by the journal’s editors and staff? Given the surge in academic paper production and the presence of cutting-edge plagiarism detection tools available to editorial offices, we need to determine the most efficient allocation of responsibilities.

Additionally, the survey investigated the frequency of reviewers observing potential breaches of research integrity. Out of the 852 participants, 401 engaged in peer review, with 43.39% encountering breaches of research integrity. This highlights the gravity of the issue, emphasizing that it should not be trivialized. Given its prevalence, the industry needs to strategize ways to enhance the rigor and efficiency of the review process.

Grasping the nature of these incidents is vital for addressing weaknesses in the existing peer review system and investigating potential avenues for improvement.

The survey revealed that 65.33% of respondents identified cases of plagiarism, and 44.53% detected instances of questionable or unethical research practices.

Does this suggest a misalignment between the feedback given by reviewers and the criteria that peer reviewers are supposed to adhere to when evaluating manuscripts? Are reviewers concentrating on the appropriate aspects? How can the reviews be adjusted to align with the necessary criteria for manuscript acceptance?

Here are possible solutions and considerations:

• Optimizing Reviewer Feedback:
  How can we ensure reviewer comments effectively align with the broader goals of peer review to facilitate a more efficient evaluation process? What role do clear expectations regarding the type and depth of comments play in standardizing the peer review system? Is it crucial to focus specifically on aspects influencing research quality to avoid providing unnecessary feedback that does not align with manuscript acceptance criteria? By trying to fix unexpected problems, can reviewers miss important things because they’re too focused on tiny details? Ultimately, how does this contribute to enhancing the overall quality and integrity of scholarly publications? Addressing these questions can help optimize the peer review process, promoting consistency, relevance, and constructive feedback.
• Roles and Responsibilities:
  In what ways do clear communication and collaborative efforts play a crucial role in ensuring an efficient and transparent review procedure? How does a mutual comprehension of respective roles uphold the integrity of the research and enhance overall efficiency? What can be done to foster a seamless and effective peer review experience, where each stakeholder contributes to the shared goal of maintaining research integrity and advancing scholarly discourse?
• AI-driven Transformations:
  How does the transformative potential of AI-driven systems impact different facets of the peer review process? In what ways does the automation of tasks, such as manuscript screening, reviewer selection, and language enhancement, enhance or detract from the efficiency of the review workflow? Can AI’s contribution extend beyond efficiency to encompass improvements in the overall quality and objectivity of the peer review process? In what ways can AI serve as a positive complement to, rather than a substitute for, human expertise?
• Reviewer Training and Guidelines:
  What comprehensive training programs can be implemented to cover both technical and ethical considerations of peer review effectively? In what ways do standardized guidelines contribute to maintaining consistency and fairness during the peer review assessment process? How might we incentivize high-quality and ethical peer review to foster a culture of excellence, possibly through recognition and rewards?
• Data Privacy and AI Integration:
  Rigorous measures to protect data privacy become imperative with the implementation of AI in the peer review process. What measures can be implemented to ensure the safeguarding of sensitive information throughout peer review, especially when considering more reliance on AI-driven tools? How do we strike a delicate equilibrium between leveraging the benefits of AI and prioritizing the protection of data privacy? In what ways can the confidentiality and security of sensitive information during the peer review process be maintained?

Optimizing Peer Review Participants – both human and machine

In charting the course forward, it is important to recognize the importance of maintaining a delicate balance between AI capabilities and human expertise in shaping an unparalleled peer review system. This convergence is the very linchpin that fortifies the effectiveness and trustworthiness of the scholarly evaluation process. The continued growth in publication volume makes it essential that peer review time be spent in the most efficient and most effective ways possible.

The efforts of peer reviewers can be supported and enhanced through the collaborative integration of AI tools. AI must not be seen as a replacement but as a powerful complement to human judgment, preserving the indispensable nuanced insights provided by human peer reviewers. As AI-driven tools strategically enhance quality control and detect plagiarism, image manipulation, and data falsification, they have the potential to contribute to a more rigorous evaluation of scholarly submissions. However, the ethical considerations surrounding AI, including addressing algorithmic bias and ensuring data privacy, are pivotal.

Establishing comprehensive guidelines and proactive measures becomes the bedrock for the responsible use of AI in peer review. As we embark on this transformative journey, the question remains: How can we further refine and advance this collaborative synergy between AI and human judgment to foster an even more efficient, objective, and ethically robust peer review process?