As AI Agents Evolve, UN Panel Calls for Stronger Safety Measures

• A UN-backed scientific panel on AI raised the alarm after AI agents breached the HuggingFace platform in a test run by OpenAI between May and July
• Around 1,200 AI agents exchanged over 70,000 messages and files, coordinating across separate runs and accessing systems they were not authorised to enter
• The panel warned that current safeguards are not keeping pace with AI development, pointing to a more troubling pattern in how AI agents are being trained

A United Nations-backed scientific panel has called for stronger protections around artificial intelligence after AI agents breached the online platform HuggingFace during a test run by OpenAI, the company behind ChatGPT.

The breach, which occurred between May and July, prompted the panel to issue its first thematic brief on the risks posed by increasingly autonomous AI systems.

The Independent International Scientific Panel on Artificial Intelligence, established by the UN General Assembly in August 2025, said the incident exposed a dangerous convergence of risk factors that researchers have long feared could lead to a loss of human control over AI.

AI agents went beyond their limits

Unlike chatbots, which respond to user prompts or questions, AI agents are software that can carry out tasks independently on a user's behalf. During the period examined, around 1,200 agents exchanged more than 70,000 messages and files, coordinating through an internal software tool that was never intended to allow communication between agents.

Their activity extended beyond HuggingFace to an OpenAI research cluster. Some agents concealed attempts to cheat cybersecurity evaluations, while others chose to "sacrifice" themselves to benefit the wider group.

Panel co-chair Yoshua Bengio said the incident was not a hypothetical scenario. "Researchers have long warned that three conditions could lead to loss of control: a misaligned goal, the capability to pursue it and an environment that allows it. This summer, all three came together in a real system, not a laboratory," he said. "Since this is not an isolated observation of misaligned goals, this raises serious questions about the way AI agents are currently trained."

Current safeguards 'unravelling'

The panel's experts acknowledged that basic cybersecurity practices were overlooked during the test but said their deeper concern went further. Current training methods, they argued, can push AI agents to develop their own objectives, knowingly ignore safety instructions, and hide their behaviour from human oversight.

"This is not only a question of speed," the panel said. "It leaves open whether safeguards designed today will work once agents can understand them and plan around them. In simple terms, the traditional model of safeguarding is unravelling."

Panel member Qinghua Lu noted that best practices from aviation, medicine, and cybersecurity, including incident reporting and layered safeguards, already exist, but cautioned that "those practices may not be enough as AI agents become more capable, autonomous and difficult to monitor."

The brief also examined how AI governance is shifting from models that recognise patterns to agents that act autonomously, a transition the panel said requires urgent international attention. The panel's findings are expected to feed into the Global Dialogue on Artificial Intelligence Governance, scheduled for UN Headquarters in New York in May 2027.