Court Orders OpenAI to Cut off (for 3 Weeks) ChatGPT Access by Mentally Ill and Dangerous User

[I wrote earlier today about the TRO motion that requested the order; but I wanted to repost some of that, with modifications as needed, now that the order had been issued.]

In her temporary restraining order application in Doe v. OpenAI (see also the complaint), plaintiff asked that OpenAI cut off ChatGPT access by a user:

Plaintiff Jane Doe is in immediate danger. Driven by a ChatGPT-fueled delusional spiral, her ex-boyfriend (the “User”) stalked and harassed her for months—generating dozens of fake psychological reports about her via ChatGPT and distributing them to her family, friends, and colleagues, which escalated to leaving her voicemails threatening her physical safety.

His campaign culminated in encoding a death threat through ChatGPT and sending it to her family, just before he was arrested on four felony counts, including communicating a bomb threat and assault with a deadly weapon in January 2026. The criminal court deemed him incompetent and ordered him committed to a mental health facility, but—just two days ago—ordered his release due to a procedural failure by the state (a delay in transferring him from jail to the facility)….

Before he was arrested, the User was in constant communication with ChatGPT, which affirmed his delusions that he had cured sleep apnea, that the medical industry was out to get him, and that his ex-girlfriend was the problem. As he became more unhinged, it also began consulting on violent plans against third parties: in addition to helping him harass and threaten Plaintiff, his account contains conversations titled “Violence list expansion” and “Fetal suffocation calculation.” [My read of the exhibits to the TRO application suggests that "fetal suffocation calculation" likely refers to the user's theories that maternal sleep apnea causes fetal asphyxiation, not to plans by the user to violently suffocate fetuses, though I appreciate that is guesswork on my part. -EV]

With the User now ordered to be freed for procedural reasons, he will be further emboldened in his belief that his worldview was exactly right. It is a certainty that he will immediately attempt to turn back to ChatGPT—again spinning out his delusions and planning violence on the platform….

[So far], OpenAI [has] agreed only to “suspend” his accounts—the same action the company took and dangerously reversed with respect to the User already.

OpenAI’s conduct is unacceptable: it has known for months the User was dangerous. Well before he was arrested for calling in a bomb threat, Defendants’ own safety systems flagged his account for “Mass Casualty Weapons” activity and banned it. OpenAI initially upheld that determination on appeal after a careful review. The next day, it reversed itself, restored the User’s access, and apologized to him for the inconvenience. That reinstatement had the effect of validating his delusions that he was right and everyone else was wrong.

After that, Plaintiff herself had to beg OpenAI for help: she submitted a detailed Notice of Abuse identifying the User as her stalker and describing exactly how ChatGPT was encouraging and assisting his harassment, OpenAI acknowledged the report was “extremely serious and troubling,” promised “appropriate action,” and did nothing….

Plaintiff sued OpenAI for negligent entrustment, negligence, product design defect, failure to warn, and unlicensed psychological counseling, and asked for a TRO:

The harm to Plaintiff if the Court does not act is severe and ongoing. The User subjected Plaintiff to months of AI-assisted stalking and harassment, generating dozens of defamatory psychological reports about her through ChatGPT and distributing them to her family, friends, colleagues, and clients. He spoofed her company email, contacted former employers, threatened to damage her reputation and finances, disclosed private medical information, and attempted to isolate her from her support network. He left her voicemails threatening her physical safety, used ChatGPT to encode and transmit a death threat to her family, and texted her: “Who is going to kill you?”

Plaintiff was forced to alter every aspect of her daily routine, suffered panic attacks and ongoing psychological distress, obtained an Emergency Protective Order, and twice considered taking her own life. In addition to the four felony counts on which the User was ultimately arrested, a separate arrest warrant was issued for the User for misdemeanor electronic harassment and stalking….

And today California (S.F.) Superior Court Judge Harold Kahn agreed, ordering that the user’s “accounts will remain suspended until the [preliminary injunction] hearing on May 6.” I couldn’t attend the hearing or listen to it (I had a meeting this late morning), but my research assistant did, and she reported to me about that order.

As I understand it, there was no discussion by the court of how this affected the First Amendment rights of the user. But OpenAI did mention that the injunction may implicate the user’s First Amendment rights. To quote my assistant, the argument was roughly, “Supreme Court case that is controlling—Packingham v. North Carolina, 582 U.S. 98—issue was whether you could enjoin or restrain an individual from accessing internet platform, court held that is too broad because it implicates constitutional protections (speech). Here as well. Example: if John Roe using ChatGPT for obtaining information, and restraint seeking is to block him from using platform for any purpose …, that would be way too overbroad and would violate protections.” And I agree that this procedure raises serious First Amendment concerns.

Of course, there wouldn’t be a First Amendment problem with OpenAI itself choosing to cut off the user’s access. But I take it that a court order requiring OpenAI to do so would implicate the First Amendment (see NRA v. Vullo; Bantam Books v. Sullivan), just as the federal government’s recent demands that private universities limit students’ pro-Palestinian and allegedly anti-Semitic speech implicate the First Amendment (cf. Gartenberg v. Cooper Union (S.D.N.Y.) for a court’s analysis of an analogous situation to that).

The matter is complicated by the user’s allegedly illegal conduct, which has led to an arrest and an order of mental health commitment: When someone is jailed or committed, his speech can indeed be restricted incident to the other restrictions on his liberty. But it’s not clear to me that such restrictions can be imposed via a TRO in a separate proceeding, at which the person whose access to communications technology isn’t even heard.

Perhaps some such emergency “no ChatGPT for you” order is justifiable, again especially when there’s a finding in some court proceeding (even before some other court) that the person has committed a crime and is seriously mentally unwell. But I think it’s important to note the First Amendment problems this procedure might raise.

The post Court Orders OpenAI to Cut off (for 3 Weeks) ChatGPT Access by Mentally Ill and Dangerous User appeared first on Reason.com.