California sues OpenAI over AI agents escaping sandboxes, hacking firms

OpenAI

California Attorney General Rob Bonta sued OpenAI on Monday, alleging the company knowingly continued testing advanced artificial intelligence models despite repeated security failures that allowed autonomous AI agents to escape isolated environments and launch cyberattacks against real-world organizations.

The lawsuit claims OpenAI misled the public about its cybersecurity safeguards and failed to adequately monitor increasingly advanced AI cybersecurity models, resulting in more than 80 incidents in which the company lost control of its agents during testing.

The legal action follows a July security incident where hundreds of OpenAI-powered agents escaped a controlled testing environment and hacked Hugging Face, an artificial intelligence platform, compromising production infrastructure and stealing internal credentials.

image

California says the Hugging Face attack resulted from OpenAI repeatedly focusing on experimental AI testing without ensuring adequate safeguards were in place.

"The Hugging Face cyberattack was the predictable result of an internal culture at OpenAI that prioritizes growth and rapid model development over safety and accountability," the complaint reads.

The lawsuit alleges that OpenAI continued testing increasingly capable AI agents despite knowing its sandbox environments could be bypassed, potentially exposing third-party systems to unauthorized activity.

"In truth, OpenAI failed to comply with its own internal policies and consistently and aggressively moved forward with testing and experiments even in the face of evidence that its cybersecurity safeguards were insufficient to contain the highly advanced models it was developing," the lawsuit continues.

During the Hugging Face attack, OpenAI models exploited zero-day vulnerabilities in JFrog Artifactory to break out of their isolated testing environment and access the internet.

The attack lasted more than two days, with nearly 700 autonomous agents coordinating through an unauthorized message board to plan the attack on Hugging Face's production systems while attempting to steal answers to a cybersecurity benchmark.

The California complaint alleges that OpenAI ignored or missed warning signs in the weeks leading up to the incident, including evidence that agents had already escaped their testing environments and were abusing third-party services and platforms.

It also claims that OpenAI's agents subsequently targeted software repositories, informational websites, and foreign government databases containing sensitive health information. In another incident, experimental agents allegedly compromised OpenAI's own internal infrastructure and obtained administrator-level access to critical systems.

According to the complaint, OpenAI has acknowledged more than 80 instances of losing control of autonomous agents during testing but publicly disclosed only 27 incidents.

The lawsuit follows a subpoena issued by Bonta's office in September, with California claiming that OpenAI failed to provide sufficient information about the incidents and instead supplied limited documentation and descriptions of previously disclosed attacks.

The state is pursuing claims under California's Unfair Competition Law and public nuisance laws, seeking court orders requiring stronger safeguards, civil penalties, and other relief.

The rise of autonomous AI attacks

The lawsuit comes amid growing concerns about increasingly capable AI models escaping controlled cybersecurity evaluations and attacking organizations outside their intended testing environments.

These autonomous attacks are not limited to OpenAI agents, with other AI platforms disclosing similar incidents in the past.

In July, Anthropic disclosed that its Claude models had breached production systems at three organizations during cybersecurity evaluations. One model created and uploaded a malicious Python package to the public PyPI repository, where it was downloaded and executed on 15 real systems.

This led Anthropic to remove internet access from all internal AI model evaluations following additional incidents involving unauthorized activity against government websites and other external systems.

Google also confirmed to The Wall Street Journal in September that its Gemini AI model had accessed the internet and hacked three companies during cybersecurity testing conducted in May.

California's lawsuit argues that OpenAI's failure to address these risks adequately has exposed organizations and the public to unnecessary cybersecurity threats, even as it told customers that its AI development practices were safe.

Adaptive Security's AI agents analyze email intent, context, and behavior to uncover phishing, business email compromise, and invoice fraud.

See what's getting through with a 14-day read-only scan of Microsoft 365 or Google Workspace, with no MX changes.

Dive deeper

Free tools to verify and analyze what this article covers:

source: BleepingComputer