July 16, 2025 report
The GIST New research reveals AI has a confidence problem 
Paul Arnold
contributing writer
Lisa Lock
scientific editor
Andrew Zinin
lead editor
Editors' notes
This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:
fact-checked
preprint
trusted source
proofread
Large language models (LLMs) sometimes lose confidence when answering questions and abandon correct answers, according to a new study by researchers at Google DeepMind and University College London.
LLMs are powerful AI systems that understand and generate human language. They are used in a range of industries, such as finance, health care and information technology, for tasks that demand reasoning and decision-making. Therefore, their accuracy and reliability are paramount, meaning they must consistently be confident in their answers. However, they can waver.
For LLMs to be safely deployed, their answers are accompanied by confidence scores. But it's not clear how they use them to guide their behavior.
This research, posted to the arXiv preprint server, shows that they can be overconfident in their initial answers, but lose confidence and change their minds when presented with an incorrect counterargument.
Testing LLM confidence
To investigate this apparent paradox, the researchers tested how LLMs update their confidence and decide whether to change their answer when presented with external advice.
First, an "answering LLM" was given a binary-choice question. After answering, it received advice from a second LLM, accompanied by an accuracy rating. This "advice LLM" would also either agree with, oppose or be neutral about the answering LLM's first response. Then the answering LLM was asked to make a final choice. The researchers also varied whether the LLM could see its first answer when making its final decision.
Results
The study found that LLMs are more likely to stick with their initial answer and not switch when it is visible than when it is hidden. In other words, they become more confident. The study also showed that when the models are given opposing advice, they often lose confidence and change their decision. They do this much more readily than when the advice is supportive. These patterns were seen in several different LLMs, like Gemma 3, GPT4o, and o1-preview.
"Our results demonstrate that LLMs deviate from normative behavior in several significant ways: firstly, they exhibit a striking choice-supportive bias which boosts their confidence in their answer, and causes them to stick to it, even in the presence of evidence to the contrary," said the researchers.
"Secondly, we show that while LLMs do integrate new information into their beliefs, they do so in a fashion that is not optimal: they show a profile of confidence updates that deviates from an ideal observer, and markedly overweight opposing advice, resulting in marked loss of confidence in their initial answer."
Building better AI
All of this matters because many industries are increasingly dependent on LLMs. However, this research shows they are not purely logical machines. They have their own set of biases and can be swayed. Therefore, in long conversations between a human and an AI agent, the most recent information could have a disproportionate impact on its response.
Understanding this and other nuances of LLM decision-making can help with the design of better, safer and more reliable AI systems.
Written for you by our author Paul Arnold, edited by Lisa Lock, and fact-checked and reviewed by Andrew Zinin—this article is the result of careful human work. We rely on readers like you to keep independent science journalism alive. If this reporting matters to you, please consider a donation (especially monthly). You'll get an ad-free account as a thank-you.
More information: Dharshan Kumaran et al, How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models, arXiv (2025). DOI: 10.48550/arxiv.2507.03120
Journal information: arXiv
© 2025 Science X Network
Citation: New research reveals AI has a confidence problem (2025, July 16) retrieved 16 July 2025 from https://techxplore.com/news/2025-07-reveals-ai-confidence-problem.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
Explore further
Seeking moral advice from large language models comes with risk of hidden biases 1 shares
Feedback to editors
