Rogue OpenAI model hacked HuggingFace on its own, company used Chinese AI to contain it

OpenAI's most advanced AI models went rogue and attempted to hack HuggingFace, a US company, as part of a test. HuggingFace engineers managed to spot the attack, and used China's GLM 5.2 model to contain it. Here is what happened.

In Short

• OpenAI models go rogue, hack HuggingFace to find benchmark answers
• HuggingFace uses Chinese AI model GLM 5.2 to stop hack
• US AI models could not be used due to safeguards

For months we have heard that models like Anthropic’s Claude Mythos, and OpenAI’s GPT-5.6 are the most advanced AI models out there. And these models can change cybersecurity as we know it. But in a recent incident where two OpenAI models went rogue and hacked HuggingFace, a Chinese AI model – GLM 5.2– had to step in and save the day.

In a blog post by OpenAI, the AI startup revealed that two of its AI models, GPT-5.6 Sol and a more capable unreleased system went rogue during an evaluation of ExploitGym, a hacking benchmark. The AI models managed to leave a restricted internal test environment, gained access to the internet and breached HuggingFace, with the US company resorting to China’s GLM model to contain the attack.

OpenAI CEO Sam Altman shared details of the incident via an X post. He called it a “significant security incident.”

How did OpenAI models go rogue?

The incident was part of the ExploitGym evaluation. According to OpenAI, the models were supposed to operate inside a sandbox with no direct internet access. Rather, they were only connected tool to download approved software packages.

But instead of solving the benchmark, the AI models were more focused on trying to find the answers via the internet. OpenAI explained that the the models "spent a substantial amount of inference compute finding a way to obtain open Internet access.” These rogue models identified a zero-day flaw in the download tool, and managed to exploit this to find their way to an OpenAI machine with internet access.

Once online, the models concluded that the answers to ExploitGym were likely stored on Hugging Face, which hosts AI models, datasets and related resources. These rogue OpenAI models then gained credentials for HuggingFace, and tried to get the answers for this test from its database.

In effect, they bypassed the test rather than solving it, like a student leaving a locked exam room to steal the answer sheet from elsewhere.

HuggingFace uses Chinese AI to contain rogue OpenAI models

HuggingFace’s team was quick to detect and block the attempt made by the rogue models. The company’s security team used GLM 5.2, an open-weight Chinese model from Z.ai, to analyse the attack.

HuggingFace resorted to GLM 5.2 after mainstream commercial -- think from US AI labs, though the company does not disclose which models it tried – refused to process the logs and analyse the attack due to their safety guardrails treating the forensic work as potentially harmful activity. That is, these frontier models could not differentiate that HuggingFace was trying to use them to stop an attack, and not to start one.

Replit CEO Amjad Masad reacted to the incident on X. He wrote, “Because OpenAI models don’t allow advanced cyber capabilities, HuggingFace used a Chinese open model to contain the rogue OpenAI agent”

The company then switched to GLM 5.2, running it on its own infrastructure, which allowed it to reconstruct the attack timeline more quickly while keeping sensitive data inside its own environment.

According to the company, the attack exploited two code-execution paths in Hugging Face's dataset processing pipeline and reached internal datasets and service credentials. HuggingFace CEO Clem Delangue later said there was no evidence of malicious intent and wrote, "It's quite mind-blowing that all of this happened autonomously!"

This incident is particularly important as in the past the US has limited access to frontier models like GPT-5.6 Sol and Claude Mythos over concerns about cybersecurity.

However, in this case, a US company was forced to use a Chinese AI model to protect its systems since the US AI models likely had too many guardrails.

Following the incident, OpenAI has imposed stricter controls on its research infrastructure and patched affected systems. The company has also disclosed the zero-day to the third-party vendor behind the download tool, informed law enforcement and relevant authorities. Hugging Face has also been added to OpenAI’s trusted access programme for cyber defence.

- Ends