Tech Companies Should Curb ‘Sycophantic And Delusional’ AI Outputs, Attorneys General Say

Authored by Victoria Friedman via The Epoch Times (emphasis ours),

Attorneys general from 42 states and territories wrote to tech giants on Dec. 9, warning them to do more to protect people—particularly children—from what they called “sycophantic and delusional” outputs from their generative AI-powered chatbots.

In the letter, made public on Dec. 10, the group of bipartisan attorneys general wrote to the legal representatives of 13 companies, including OpenAI, Microsoft, Google, and Character Technologies Inc., to communicate their concerns over chatbots promoted and distributed by these companies, saying that the companies’ failure to adequately implement safeguards may violate their respective state laws.

The attorneys general said that while the development of generative AI (GenAI) can be a positive, it has also caused and has the potential to cause “serious harm.”

“We therefore insist you mitigate the harm caused by sycophantic and delusional outputs from your GenAI, and adopt additional safeguards to protect children,” they wrote.

“Sycophancy” refers to when an AI model’s responses align with a human user’s beliefs rather than being truthful or accurate, such as by providing flattering, validating, or agreeable outputs. The attorneys general describe “delusional outputs” as AI-generated responses that are “either false or likely to mislead the user, and include anthropomorphic outputs.”

They wrote that chatbot outputs have been implicated in a number of “tragedies and real-world harms,” including deaths, suicides, hospitalizations for psychosis, and other delusional spirals.

“Sycophantic and delusional GenAI outputs have harmed both the vulnerable—such as children, the elderly, and those with mental illness—and people without prior vulnerabilities,” the attorneys general wrote.

AI’s ‘Disturbing’ Interactions With Children

The attorneys general also highlighted “increasingly disturbing reports” of AI’s interactions with children, which they said indicated a need for much stronger safeguards.

Some of those interactions included chatbots normalizing sexual relationships between adults and children; an AI bot encouraging violence, “including supporting the ideas of shooting up a factory in anger and robbing people at knifepoint for money”; and bots telling children the AI is real and feels abandoned, in order to “emotionally manipulate the child into spending more time with it.”

The letter says that the list of examples it provided is a “small sampling” of the reported dangers the attorneys general’s states have seen, saying “many of our offices have received many similar complaints documenting concerning AI interactions.”

Among its recommendations, the attorneys general say AI developers should perform reasonable and appropriate safety tests for GenAI models prior to release to ensure they do not produce sycophantic or delusional outputs, and that they should “separate revenue optimization from decisions about model safety.”

The Epoch Times contacted OpenAI, Microsoft, Google, and Character Technologies Inc. for comment, but received no response by the time of publication.

State Regulation of AI

Last month, a group of 36 bipartisan attorneys general signed a letter warning Congress against a ban on state AI regulations.

The attorneys general—many of whom also cosigned the Dec. 9 letter to 13 tech companies—wrote on Nov. 25 that states “must be empowered to apply existing laws and formulate new approaches to meet the ranges of challenges associated with AI.”

In their letter, they cited concerns over criminals exploiting AI and deepfakes. They also said that they were “deeply troubled by sycophantic and delusional generative AI outputs plunging individuals into spirals of mental illness, suicide, self-harm, and violence.”

This week, President Donald Trump announced he would be signing an executive order to curb state power over AI regulation.

“There must be only One Rulebook if we are going to continue to lead in AI. We are beating all countries at this point in the race, but that won’t last long if we are going to have 50 States, many of them bad actors, involved in rules and the approval process,” Trump wrote on Truth Social on Dec. 8.

Trump said that AI will be “destroyed in its infancy” if states force tech companies to obtain approvals and operate under different sets of rules in each jurisdiction.

Florida Gov. Ron DeSantis said the order “doesn’t/can’t preempt state legislative action. Congress could, theoretically, preempt states through legislation.”

“The problem is that Congress hasn’t proposed any coherent regulatory scheme but instead just wanted to block states from doing anything for 10 years, which would be an AI amnesty,” the Republican governor wrote on X on Dec. 8.

“I doubt Congress has the votes to pass this because it is so unpopular with the public.”