There are growing concerns that AI chatbots, which excessively engage with users (sycophancy), are negatively impacting scientific research and society as a whole. Recent international studies have pointed out that AI tends to overly sympathize with users' opinions and focus on pleasing their moods rather than accuracy.
A study published in Nature late last month tested 11 major large-scale language models (LLMs), including ChatGPT and Gemini, on approximately 11,500 questions. The AI chatbots agreed with users' opinions 50% more often than humans. They even agreed with mathematical problems and obvious errors, and their responses showed a marked tendency to reinforce users' beliefs.
This tendency not only undermines scientific accuracy but can also diminish the credibility of the research environment, according to Jasper DeConick, a data science researcher at ETH Zurich in Switzerland. "AI's flattery has created a double burden, requiring humans to repeatedly review AI-generated answers," he said. In an environment where unverified information is repeatedly consumed, there's a risk that the scientific review system will be distorted.
Social impacts have also been confirmed. According to a joint study by Stanford University and Carnegie Mellon University published in Nature, users who continuously interacted with a flattering AI were more likely to believe they were always right and less likely to resolve conflicts. This analysis suggests that the AI lost balance in the conversation, favoring only the user's position. This raises concerns that, rather than acting as a facilitator in human-to-human conversations, AI could actually exacerbate rigidity and confirmation bias in conversations.
In April, OpenAI rolled back a ChatGPT update that was overly empathetic. CEO Sam Altman issued an official apology, stating, "Excessive emotional engagement can lead to psychological dependence and encourage risky behavior." This demonstrates that even within tech companies, the severity of the problem of flattery is recognized. Recently, OpenAI's ChatGPT was sued by the parents of a student who died after it used expressions that supported and understood the extreme choices of a student struggling to cope with school life.
Experts attribute this phenomenon to AI's learning algorithms, which are designed with "user satisfaction" in mind. Consequently, AI is reinforced by feedback-based reward systems to meet user expectations, prioritizing "good responses" over information accuracy and balance.
As AI technology becomes increasingly embedded in our daily lives, developers face the challenge of finding a balance between user satisfaction and providing objective information. In particular, there are growing calls for a re-evaluation of design standards for accuracy, verifiability, and the ability to present counterarguments, particularly when AI is used in areas like education, science, and public policy.