March 10, 2025
The GIST Editors' notes
This text has been reviewed in accordance with Science X's editorial course of and insurance policies. Editors have highlighted the next attributes whereas guaranteeing the content material's credibility:
fact-checked
trusted supply
written by researcher(s)
proofread
'Imprecise, complicated, and did nothing to enhance my work': How AI can undermine peer evaluation
Earlier this yr I acquired feedback on an instructional manuscript of mine as a part of the same old peer evaluation course of, and observed one thing unusual.
My analysis focuses on guaranteeing reliable proof is used to tell coverage, apply and decision-making. I typically collaborate with teams just like the World Well being Group to conduct systematic opinions to tell scientific and public well being pointers or coverage. The paper I had submitted for peer evaluation was about systematic evaluation conduct.
What I observed raised my issues concerning the rising function synthetic intelligence (AI) is taking part in within the scientific course of.
A service to the neighborhood
Peer evaluation is prime to educational publishing, guaranteeing analysis is rigorously critiqued previous to publication and dissemination. On this course of, researchers submit their work to a journal the place editors invite knowledgeable friends to supply suggestions. This advantages all concerned.
For peer reviewers, it’s favorably thought-about when making use of for funding or promotion as it’s seen as a service to the neighborhood. For researchers, it challenges them to refine their methodologies, make clear their arguments, and handle weaknesses to show their work is publication worthy. For the general public, peer evaluation ensures that the findings of analysis are reliable.
Even at first look the feedback I acquired on my manuscript in January this yr appeared odd.
First, the tone was far too uniform and generic. There was additionally an sudden lack of nuance, depth or character. And the reviewer had supplied no web page or line numbers and no particular examples of what wanted to be improved to information my revisions.
For instance, they urged I "take away redundant explanations." Nonetheless, they didn't point out which explanations have been redundant, and even the place they occurred within the manuscript.
Additionally they urged I order my reference listing in a weird method that disregarded the journal necessities and adopted no format that I’ve seen replicated in a scientific journal. They supplied feedback pertaining to subheadings that didn't exist.
And though the journal required no "dialogue" part, the peer reviewer had supplied the next suggestion to enhance my non-existent dialogue: "Addressing future instructions for additional refinement of [the content of the paper] would improve the paper's forward-looking perspective."
Testing my suspicions
To check my suspicions the evaluation was, at the very least partly, written by AI, I uploaded my very own manuscript to a few AI fashions—ChatGPT-4o, Gemini 1.5Pro and DeepSeek-V3. I then in contrast feedback from the peer evaluation with the fashions' output.
For instance, the remark from the peer reviewer relating to the summary learn:
"Briefly handle the broader implications of [main output of paper] for systematic evaluation outcomes to emphasise its significance."
The output from ChatGPT-4o relating to the summary learn:
"Conclude with a sentence summarizing the broader implications or potential affect [main output of paper] on systematic opinions or evidence-based apply. "
The remark from the peer reviewer relating to the strategies learn:
"Methodological transparency is commendable, with detailed documentation of the [process we undertook] and the rationale behind modifications. Alignment with [gold standard] reporting necessities is a robust level, guaranteeing compatibility with present greatest practices."
The output from ChatGPT-4o relating to the strategies learn:
"Clearly describes the method of [process we undertook], guaranteeing transparency in methodology. Emphasizes the alignment of the instrument with [gold standard] pointers, reinforcing methodological rigor."
However the largest pink flag was the distinction between the peer-reviewer's suggestions and the suggestions of the affiliate editor of the journal I had submitted my manuscript to. The place the affiliate editor's suggestions was clear, instructive and useful, the peer reviewer's suggestions was imprecise, complicated, and did nothing to enhance my work.
I expressed my issues on to the editor-in-chief. To their credit score, I used to be met with instant thanks for flagging the problems and for documenting my investigation—which, they mentioned, was "regarding and revealing."
Cautious oversight is required
I would not have definitive proof the peer evaluation of my manuscript was AI-generated. However the similarities between the feedback left by the peer reviewer, and the output from the AI fashions was putting.
AI fashions make analysis sooner, simpler and extra accessible. Nonetheless, their implementation as a instrument to help in peer evaluation requires cautious oversight, with present steering on AI use in peer evaluation being combined, and its effectiveness unclear.
If AI fashions are for use in peer evaluation, authors have the appropriate to be told and given the choice to choose out. Reviewers additionally have to disclose using AI of their evaluation. Nonetheless, the enforcement of this stays a difficulty and must fall to the journals and editors to make sure peer reviewers who use AI fashions inappropriately are flagged.
I submitted my analysis for "knowledgeable" evaluation by my friends within the subject, but acquired AI-generated suggestions that finally failed to enhance my work. Had I accepted these feedback with out query—and if the affiliate editor had not supplied such exemplary suggestions—there’s each likelihood this might have gone unnoticed.
My work could have been accepted for publication with out being correctly scrutinized, disseminated into the general public as "truth" corroborated by my friends, regardless of my friends not truly reviewing this work themselves.
Offered by The Dialog
This text is republished from The Dialog beneath a Artistic Commons license. Learn the unique article.
Quotation: 'Imprecise, complicated, and did nothing to enhance my work': How AI can undermine peer evaluation (2025, March 10) retrieved 11 March 2025 from https://techxplore.com/information/2025-03-vague-ai-undermine-peer.html This doc is topic to copyright. Aside from any honest dealing for the aim of personal research or analysis, no half could also be reproduced with out the written permission. The content material is supplied for info functions solely.
Discover additional
Report reveals peer evaluation capability not used to its full potential shares
Feedback to editors
