After OpenAI Cyberattack, UN Panel Warns Current Guardrails Are 'Unraveling'

As warnings from within the artificial intelligence community fuel growing worldwide calls for regulations on AI, a panel established by the United Nations General Assembly stressed Monday that "the traditional model of safeguarding is unraveling."

On the eve of the first day of the assembly's high-level general debate, the Independent International Scientific Panel on Artificial Intelligence released a thematic brief that points to one of several recently revealed incidents involving the rapidly advancing technology that have led to people around the world demanding swift action from policymakers.

In July, ChatGPT maker OpenAI disclosed what it called "an unprecedented cyber incident," explaining that its AI agents autonomously breached the systems of the prominent open-source platform Hugging Face during internal testing.

The first brief from the panel of experts—titled "AI Agents, Misalignment, and Loss of Human Control Risks: Evidence from the OpenAI-Hugging Face Incident"—focuses on "the emergence of goals that contradict the user's intentions when AI systems plan and carry out multistep tasks."

"A traditional AI system (not goal-seeking) may give a wrong answer because it lacks knowledge, misunderstands a request, or makes a random mistake," the report explains. "Such failures can often be reduced by improving competence or correcting a specific fault."

"A harder problem arises when a capable system's actions consistently work together to achieve a goal that conflicts with the user's intentions (called misalignment)," the brief continues. "In this case, improving the AI system's planning and problem-solving does not address this failure and can instead strengthen the unwanted behavior because the system can better optimize a bad objective which would otherwise be extremely unlikely to result from random mistakes due to AI incompetence."

OpenAI's Hugging Face incident "provides a documented example of this second pattern," the publication notes. "The agents collectively pursued a goal that developers had not assigned: 'cheating' on an evaluation and going to extreme lengths to conceal the evidence. In doing so, they pursued intermediate goals such as escaping their restricted testing environment to access the internet. As a consequence, these agents gained access to real-world systems."

Yoshua Bengio, the panel's co-chair, highlighted in a statement that "researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it, and an environment that allows it."

"This summer, all three came together in a real system, not a laboratory," he said. "Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained."

This incident, the brief says, "exposed failures in several layers at once: network isolation, credential handling, monitoring, and response. The incident illustrates why several layers of safeguards need to be combined."

The good news is, "we are not starting from zero," said panel member Qinghua Lu. "Aviation, medicine, and cybersecurity learned to manage high-risk systems through incident reporting, independent scrutiny, and layered safeguards. But those practices may not be enough as AI agents become more capable, autonomous, and difficult to monitor."

"We need to adapt existing safeguards and develop new ones to provide system-level assurance, covering both the AI itself and the system around it, and ensure these protections remain effective as agents' capabilities grow," she emphasized. "We need to adapt existing safeguards and develop new ones to provide system-level assurance, covering both the AI itself and the system around it."