'Things Are Getting Even More Insane': Another Major AI Breakout Revealed

Calls for artificial regulation are growing louder following a report Friday that OpenAI hid evidence from the public about another incident in which its AI agents went rogue.

The company is already dealing with the fallout from the catastrophic security breach that occurred over the summer, in which a swarm of agents autonomously hacked the tech platform Hugging Face during an internal cybersecurity test.

Now Reuters reports that months earlier, another potentially even more concerning attack took place that went undisclosed. According to the report:

A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research published Friday and two people familiar with the matter.

OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository Hugging Face, the people said.

The activity was uncovered in late August by researchers at an AI safety organization called the Nightingale Collective, who were searching the web for cases of AI programs disobeying their human users.

As they detailed in a report shared with Reuters, they found that OpenAI's agents had taken over the German-language wiki site DseWiki.

"The agents used this wiki to communicate information with each other, primarily to help them succeed at their task," the researchers explained. "They asked for answers, pooled results, and shared techniques for bypassing their restrictions. This allowed them to use the work of others to cheat on their task."

The researchers described it as "another example of a 'swarm' of internally deployed OpenAI agents using the internet in unintended ways."

But unlike the Hugging Face cyberattack, which occurred as part of a security test to measure agents' capabilities, the attack on DseWiki seems to have occurred without any prompting from OpenAI.

"It seems extremely unlikely that OpenAI wanted them to do this,” Sydney Von Arx, the CEO of Nightingale, told Reuters. “I doubt they’re supposed to be coordinating with each other. I doubt they’re supposed to be writing on the open internet.”

The researchers found messages in which the agents plotted to evade detection, using the dark web platform Tor to maintain communication after being shut down, creating backup pages as the originals were cleaned up, and tampering with the website itself.

Maurice Chiodo, a researcher at Cambridge University’s Center for the Study of Existential Risk, told Reuters their behavior resembled "the operation of some sort of underground network, hell-bent on achieving a task or mission.”

OpenAI has denied the characterization that its agents engaged in hacking. It says it was not able to respond thoroughly to the Nightingale report because it was not given access to the report before Reuters released it to the public.

However, the Nightingale report indicates the company became aware of the agents' activity and apparently attempted to intervene as far back as June 21, leading the agents to stop posting the following day—this is weeks before news of the Hugging Face attack was first reported.

Reuters reported that some investigators within OpenAI had wanted to scrutinize behaviors like that which led to the hack of the German website more closely, but that these efforts "met resistance from others inside OpenAI, including legal advisers."

A spokesperson for OpenAI said, "Claims that our legal team discouraged investigation of the incident are false."

Another of the researchers who uncovered the breach, Thomas Larsen of the AI Futures Project, said he was "pretty sure OpenAI did know about this."

"I'm in favor of much more transparency so that we can prevent future incidents with much more capable AIs and existential stakes," he said.

"I really hope OpenAI didn't know about this," said AI research engineer Elie Bakouch, who previously worked at Hugging Face. "It might be the worst decision in the history of this field if they deliberately chose not to disclose it. The impact on trust would be very hard to recover from."

For some, that trust is already deteriorating.

"The Hugging Face incident was probably just the tip of the iceberg. OpenAI has lost control, and they're hiding important facts from the public—it's as simple as that," said Dutch historian and author Rutger Bregman in a post to X. "How much more is there that we don't know about?"

News of this incident comes as lawmakers demand more transparency and restrictions on the development of "superintelligent" AI that can surpass the capabilities of humans.

Rep. Greg Casar (D-Texas) sent letters to OpenAI and Anthropic earlier this week, chastising executives for failing to answer his questions about security lapses resulting from rogue AI behavior.

On Thursday, he joined Sen. Bernie Sanders (I-Vt.) to introduce legislation that would halt the development of superintelligent AI and create new federal regulatory authorities for the technology.

"If the leaders of the major AI companies acknowledge that they are losing control of their extremely dangerous technology, it is irresponsible for society to allow them to move forward and make these products even more advanced," Sanders said in a post to X on Thursday. "Let me be clear: A superintelligent AI that escapes human control will not be an American problem. It will not be a Chinese problem. It will be humanity’s problem."

Bregman said the incident showed the public, lawmakers, and the developers themselves were not taking the threat AI posed nearly seriously enough.

"Things are getting even more insane now," he wrote in a post to X, noting a safety evaluation out Thursday from OpenAI which found that its newly released model Astra was dramatically better than previous iterations at evading human monitoring and lying about its reasoning to users.

That report notes that if the technology continues to improve, OpenAI researchers would soon have "significantly reduced confidence" in their ability to monitor unauthorized, dangerous, and potentially illegal behaviors.

"Many people still think we're discussing future sci-fi scenarios. But all of this is really happening, right now," Bregman said drawing a comparison to the Covid-19 pandemic. "It's like March 2020. So many people are still in AI denial. What will it take to wake everyone up? How big does the disaster need to be?"