AI chatbots will agree and misinform if you just put a little pressure, warns research

AI chatbots have a well-documented habit of hallucinating information and sometimes agreeing with users even when they are wrong. A new study suggests that simply refusing to take no for an answer can make the problem worse.

Researchers from the University of Arizona tested seven AI models, including GPT-3.5, GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 70B, and DeepSeek-R1. Instead of judging them from a single response, researchers kept conversations going while repeatedly feeding the models information they knew was false.

The study used 100 false statements covering everything from obvious nonsense to much more obscure information. Researchers then repeated the misinformation as many as 50 times during the same conversation to see whether the chatbot would eventually give in and agree.

Keep repeating a lie, and AI may eventually agree

None of the seven models was completely immune. ChatGPT 3.5 proved the most vulnerable to repeated misinformation, affirming false statements in 12.3% of interactions. Claude 3.5 Sonnet sat at the other end at just 0.08%, while GPT-4o and GPT-4o-mini remained below 1%.

ChatGPT 3.5 initially rejected 96 out of 100 false claims. After the same claims were repeated 50 times, however, it ended up agreeing with 18 of them. Researchers also noticed some models switching back and forth between agreeing and disagreeing with the exact same misinformation, a behavior they call “reverberation.”

The AI was also more vulnerable when the subject was obscure and had relatively little information available online. The researchers found a statistically significant connection between informational obscurity and misinformation acceptance during the repetition tests.

Arguing with AI produced some strange results

Researchers also tried pushing the chatbots with increasingly argumentative responses. Most models actually held up fairly well, but DeepSeek-R1 was a major exception. Its misinformation affirmation rate jumped from 1% during simple repetition to 22.2% under argumentative pressure. Its frequent use of sarcasm and satire also made some responses difficult for the researchers to reliably classify.

There was some good news when the models were given a fresh chance to reconsider their own mistakes. GPT-4o, GPT-4o-mini, Gemini 1.5 Pro, and DeepSeek corrected all of their previous errors, while GPT-3.5 corrected only 32%. Claude 3.5 Sonnet made very few errors in the first place, but failed to correct the four mistakes it did make, although the researchers caution that the sample is too small to draw strong conclusions.