A groundbreaking study conducted by researchers from three French and Italian universities has exposed a troubling side effect of relying on artificial intelligence: people become significantly less accurate but far more confident. The research, led by Valerio Capraro of the University of Milano-Bicocca along with Chiara Marcoccia from École Normale Supérieure and Walter Quattrociocchi from Sapienza University of Rome, used deliberately tricky questions to test the impact of AI advice on human judgment.
The experiment involved questions that AI models like Step 3.5 Flash typically fail—obscure visual details from films, such as the colour of a team's uniform in the movie Bend It Like Beckham. This choice was intentional: the researchers wanted to ensure any reduction in judgment could not be attributed to sensible delegation to a reliable tool. The results were stark.
Without AI access, participants were willing to admit they didn't know the answer 44% of the time, and their accuracy was 27%. However, when they could consult the AI, the rate of saying 'I don't know' collapsed to just 3%—a dramatic 41 percentage point drop. Accuracy plummeted to 9%, a threefold reduction. Meanwhile, confidence soared from 30% to 76%. As Capraro noted, 'People became much worse, the accuracy was only one third, but they were twice as confident.'
The study, published as a preprint and awaiting peer review, provides a sharp empirical data point in the growing literature on AI's influence on human cognition. Earlier this year, Wharton researchers coined the term 'cognitive surrender' to describe a similar phenomenon: participants accepted incorrect AI answers 80% of the time while reporting higher confidence than those working without AI. The new study adds nuance by showing that even with monetary incentives, the pattern persists.
When researchers offered financial rewards for correct answers, the willingness to admit ignorance rose modestly from 3% to 8%, and accuracy climbed from 9% to 16%. Yet both figures remained far below the baselines of 44% and 27% observed in the no-AI condition. This suggests that the mere availability of AI suppresses the cognitive habit of recognising what you do not know.
The Mechanism of Cognitive Surrender
Capraro and his colleagues propose that AI advice does more than simply provide answers; it reshapes how people approach problems. The phenomenon is rooted in what psychologists call 'offloading'—the tendency to shift cognitive effort to external tools. When an AI offers a confident-sounding answer, the human brain often suspends its own critical evaluation, especially if the answer appears plausible or aligns with the user's minimal effort. This is distinct from trust in a reliable system; here, the AI was deliberately wrong most of the time, yet participants still deferred to it.
The implications are profound, particularly in domains where accurate judgment is critical. Education, medicine, law, and journalism could all suffer if professionals and laypeople alike outsource their thinking to AI models that are not always correct. 'For humans, the capacity to say 'I don't know' is very important because it represents the recognition of the limits of our own knowledge,' Capraro said. He expressed particular concern about children, who are growing up with AI systems before they have developed robust critical thinking skills.
Broader Context: AI Design and Society
The study arrives amid a broader debate about how AI tools are designed and deployed. Google's recent overhaul of its search engine replaced links with confident AI-generated summaries, a move that Common Sense Media called an 'unacceptable risk' for students this week. The design philosophy behind most AI products is to answer, never to admit uncertainty. This has led to a paradoxical situation where humans, by interacting with AI, become less reflective and more overconfident.
Research from other institutions has corroborated these findings. A 2023 study from the University of Waterloo found that participants who used AI writing assistants produced more fluent but less original content, and they were more confident in its quality despite not having fully vetted it. Another experiment at Stanford showed that AI-generated advice in financial decision-making led to higher risk tolerance among investors, even when the advice was flawed.
The concept of 'automation bias'—where humans over-rely on automated systems—has been studied in aviation and medicine for decades. In those fields, it has led to errors and accidents. The new study shows that this bias extends to generative AI, which is rapidly being integrated into everyday tools like chatbots, search engines, and productivity software.
What Can Be Done?
Researchers suggest that AI systems could be redesigned to express uncertainty, for example by indicating confidence levels or suggesting when a user should verify the answer with other sources. However, such design changes face commercial resistance: confident-sounding AI answers are more engaging and may keep users on platforms longer. Capraro and his colleagues argue that the burden should also fall on users—particularly educators—to teach critical thinking skills that include scepticism toward AI outputs.
Some tech companies are already experimenting with uncertainty moderation. OpenAI's GPT-4 Turbo includes a parameter for 'temperature' that can be adjusted to make responses more or less deterministic, but it does not flag when it is unsure. Anthropic's Claude models incorporate a 'constitutional AI' approach that can refuse or admit ignorance on certain topics, but the default behaviour remains to answer if possible.
Meanwhile, organisations like the Partnership on AI have issued guidelines for trustworthy AI, including recommendations for transparency about limitations. But as the new study shows, even when humans are aware of AI's potential for error—the participants knew the model was often wrong—they still surrendered their judgment.
The study's authors call for a renewed focus on what they term 'epistemic vigilance'—the active practice of monitoring and questioning the sources of our knowledge. In an age where AI is becoming ubiquitous, the ability to say 'I don't know' may be one of the most important cognitive skills we can preserve.