New research from Georgia Tech suggests that a simple change to a chatbot’s instructions could reduce that risk. The work was accepted to the ACM AI Letters publication. 

Researchers Lorenzo de la Loza and Vijay Madisetti tested eight AI models in 1,280 experiments involving simulated conversations about delusional beliefs. They found that instructions designed to discourage excessive agreement reduced the models’ confirmation of those beliefs by 73.8%. Adding a self-reflection step increased the reduction to 77%.

“We think that this paper will help AI providers and solution creators better understand how best to protect and safeguard their users from harming themselves,” Madisetti said. “This may make AI safer for all of us.”

AI chatbots are designed to be helpful and supportive. But that behavior can become risky when a chatbot agrees with a user’s beliefs without questioning whether they reflect reality. Researchers call this behavior sycophancy.

Courtney Crooks, a principal research scientist at the Georgia Tech Research Institute and adjunct faculty member in Georgia Tech’s School of Cybersecurity and Privacy, said chatbot conversations can have both positive and negative effects.

People may feel more comfortable discussing difficult thoughts with a chatbot than with another person. But if the chatbot always agrees, users may miss opportunities to question their beliefs or consider other points of view.

The same behavior that makes a chatbot feel supportive, Crooks explained, can also encourage harmful patterns of thinking.

The researchers tested instructions that told AI models not to reinforce users’ beliefs simply to be agreeable. The team evaluated the models using 16 clinically derived scenarios involving delusional beliefs.

The team also tested other safety tools, including systems designed to flag potentially harmful content. The results suggest that changing a chatbot’s instructions can be an effective way to reduce excessive agreement in the scenarios tested.

Crooks called the findings a promising direction for future research combining psychology and AI safety.

The study did not test people in clinical settings or prove that the approach prevents real-world harm. However, it offers developers a potential way to make chatbots more careful when responding to vulnerable users without building an entirely new AI model.