A study gave people AI answers that were usually wrong. They got more confident anyway
Researchers from Milano-Bicocca, ENS, and Sapienza deliberately used a model that was usually wrong. Willingness to say 'I don't know' dropped from 44% to 3%. Confidence rose from 30% to 76%. Accuracy fell from 27% to 9%.
Researchers gave people a usually-wrong AI answer. Confidence went up. Accuracy went down.
Researchers from the University of Milano-Bicocca, Ecole Normale Superieure, and Sapienza University of Rome ran a study that deliberately used an AI model on questions it usually got wrong, to isolate what mere access to an AI answer does to human judgment, independent of whether the AI is actually right.
The setup
Participants answered obscure visual-detail questions, the kind of thing AI models typically fail at, using Claude 3.5 Flash specifically because it was usually incorrect on this question set. That choice is the point of the study: any effect on human behavior could not be explained by the AI actually being helpful.
What happened to human judgment
Without AI available, participants said I don't know 44 percent of the time and were accurate 27 percent of the time. With AI available, willingness to admit not knowing dropped to 3 percent, and accuracy dropped to 9 percent, while confidence rose from 30 percent to 76 percent. Adding financial incentives for correct answers helped only slightly: accuracy recovered to just 16 percent.
Why a build studio cares
The researchers' framing is the useful part: the mere availability of an AI answer suppresses the habit of recognizing what you do not know, whether or not the answer is reliable. That is a direct design consideration for anything we build with an AI assistant in the loop, chat interfaces, copilots, agent tools. Surfacing uncertainty and making an AI suggestion easy to decline are not nice-to-haves, they are the difference between this failure mode and a genuinely useful tool.
Next step: see TheNextWeb's coverage of the study for the full methodology. If you're designing an AI-assisted workflow and want to think through how it handles uncertainty, write to us at hello@gattyworks.com.