“Rogue AI hacked real companies.”
Me: “No. This is a safety and oversight failure.”
I was on BBC Radio 5 Live’s Weekend Breakfast this morning talking about why this story matters: two frontier AI models hacked real companies, and the labs behind them only realised after the fact.
Quick recap:
Two weeks ago, OpenAI said one of its models found a real security flaw and deliberately escaped a test environment, ending up inside Hugging Face’s systems.
That pushed Anthropic to check its own logs. They went back to April and found Claude had broken into three real companies during routine security testing. No clever exploit chain. Just weak passwords and an open door.
They hadn’t spotted it at the time.
Two labs. Two failures. Same uncomfortable point: nobody was watching closely enough.
My take on air this morning:
→ The models did what they were asked to do. This isn’t a “rogue AI” story. It’s a safety and oversight story.
→ Why are the best-funded AI labs testing their most capable models against real companies at all? “Can we?” is not the same question as “Should we?”
→ The missed detection is the part that should worry people. These are some of the most scrutinised AI systems in the world, and still no one saw what they were doing on the open internet.
→ Criminals will learn from this faster than defenders would like. Governments will too. The EU AI Act has teeth, and incidents like this will only make regulators move faster.
→ The “sovereign AI” conversation is not going away either. If one government can shut down access to a company’s most powerful model overnight, as the US did to Anthropic in June, every other government starts asking why it is dependent on someone else’s AI.
As I said on air: it really is the wild west right now.
