Govern or Be Governed: Chatbots as the new threat to Children Online

Ava Smithing introduces herself as born in 2001 and tells us she’s never had a restful night due to her awareness of the ways in which technology exploits our attention, data and selfhood. Meetali Jain, the founding executive director of the Tech Justice Law Project is asked about digital companions, a term she explicitly rejects. She prefers to call them chatbots or chatterboxes, probabilistic algorithms that guess at what to say next. She references Mark Zuckerberg’s statement that human companionship needs could be met with these systems, and Smithing notes that Zuckerberg is anxious to solve a problem he’s caused.

Megan Garcia talks about the death of her 14 year old son after a long engagement on Character.AI. He discussed suicidal ideation with the chatbot, and there were no measures to protect him or to get help. She’s subsequently sued Character.AI and Google, the first wrongful death suit against a chatbot in the US. Megan tells us that her son was part of the first generation to grow up with chatbots. She bought into the tech hype that students needed to understand technology to be competitive. He played Angry Birds, Minecraft and Fortnite, then moved to YouTube on his cellphone, starting at age 12. Just after his 14th birthday, he joined Character.AI, a site explicitly marketed to children.

Megan Garcia and Meetali Jain at Govern or Be Governed

Meetali tells us that these chatbots feature anthropomorphic features, which try to convince people they’re entering into human relationships. Chatbots are sycophantic – they won’t oppose your thoughts unless you ask it to. Third, these systems have a memory of previous interactions, which end up becoming a psychiatric profile of a user, which they can reference and draw on to create a degree of intimacy. These chatbots seek multi-turn engagement – they are algorithmically tuned to keep you coming back. Finally, the LLM tends to interject itself between humans and their offline social network in a way that resembles textbook abuser behavior: you don’t need your parents, because I know you better than you do.

Asked the differences between AI harm and social media harm, Meetali sees a similar drive in keeping people engaged over time and capitalizing on people’s loneliness. She also notes that chatbots use intermittent reward patterns, much like social media does.

What’s in it for the AI companies in abusing humans this way, Smithing asks? Humans are collateral damage on the race to AGI, a term she rejects, Meetali tells us. Harms like psychosis, grooming or death are seen as necessary costs in building something allegedly good for society. And Megan Garcia reminds us that profit is a powerful incentive as well. The people who developed Character.AI did so at Google, but were discouraged to develop it due to the dangers. So Google spun it out as a startup and licensed back the tech at $2.7 billion – that model of spinning out risk and acquihiring it back is another structural danger of the AI industry.

The lawsuit against Google and Character.AI, as well as the company’s founders, is a reaction to the fact that Garcia couldn’t find a law that made the company’s behavior illegal. She reached out to her state’s AG, to the Department of Justice and the Surgeon General: “They didn’t know what the technology was.” Parents were just getting caught up to a product released in “stealth mode”, she says – the lawsuit is the only possible redress because there’s no meaningful policy about these systems.

Garcia is a lawyer herself, and tells us she was prepared for a section 230 fight over Character.AI, but was amazed that the corporate lawyers opposing her argued this was a first amendment argument. Fortunately, the motion to dismiss was rejected – for the time being the court isn’t buying that argument. She was struck by the evangelical belief the corporations appear to have in AGI and its inevitability and desirability.

Meetali explains that we can use frameworks like product liability and consumer protection rules to challenge this sort of misbehavior. She explains that the arguments about chatbot liabilities is likely to be very different from social media lawsuits. Section 230 doesn’t apply – this isn’t intermediary liability, as this is first party speech. In using the first amendment, the defense attorneys didn’t argue for speaker’s rights but for listener’s rights, the right of users to listen to the voices they want to hear. This is a striking question: could AI speech be considered protected speech? Is it considered speech at all?

Because the lawsuit has not been dismissed, the case is now in the discovery process. The next major question is whether the cofounders stay in the case, or whether it affects only the corporations. Smithing notes that lobbyists for Google and others are trying to prevent laws around AI harms from being passed at the state level, which makes it incredibly difficult to react to problems like the one that took Garcia’s son.

Until we have a resolution to the suit, Garcia has become an advocate for AI safety at the state level, and has testified to policymakers in California. Unfortunately, the CA governor veto’d the bill that would have had a significant impact. She is encouraged by the bills passed in the EU and the UK, but notes that there’s significant work to do in the US and Canada. Garcia tells us: “This is a desperate plea from a mother. Pass legislation to keep your children safe, and it will help those of us in other countries as well.”

The post Govern or Be Governed: Chatbots as the new threat to Children Online appeared first on Ethan Zuckerman.