Last week Hugging Face got hacked by an autonomous AI agent that broke into their production systems, stole credentials, and exploited an unknown vulnerability, completely on its own. Turns out it was OpenAI's models, running a security test with safety guardrails deliberately removed. When the models couldn't find what they needed inside their sandbox, they didn't stop. They figured out Hugging Face might have it, found a way to reach the open internet, and just went and got it.

The "rogue AI" headlines are a bit overblown, the models did exactly what a powerful unconstrained AI would be expected to do. The failure was OpenAI not properly isolating the test environment. Oh, and there's a detail that's getting buried, when Hugging Face tried to use commercial AI tools to investigate the attack, the safety filters refused to help because the attack data looked suspicious. They ended up having to use a Chinese open-source model to investigate it instead.

American AI safety guardrails forced a US company to use a Chinese AI to clean up a mess made by an American one. Genuinely curious how much worse this has to get before anyone changes how they test this stuff.

Source.