When the first AI agents were said to have escaped their restraints and hacked a third party just two months ago, there was scepticism that perhaps AI companies like OpenAI were trying to show off the capabilities of their frontier models.
One by one, the other big AI companies then came forward to say their leading-edge models were also trying or had managed to jailbreak from their testing confines and targeted other organisations.
This week, however, comes evidence of how serious these threats were. It appears AI companies had not promptly shared or were not able to promptly share what had happened with their rogue AI agents.
Most concerning was the Australian government saying yesterday that a website had been targeted by a rogue OpenAI agent.
The agent had infiltrated a statistics portal containing “non-sensitive” data from Australia’s universal healthcare scheme, according to Prime Minister Anthony Albanese.
The Aussie leader said he had a “very frank discussion” with OpenAI boss Sam Altman for taking “too long” to disclose the breach and said there would be “legal consequences”, the BBC reported.
OpenAI found the breach in August but only sent an email to a general inbox of an Australian government agency on September 10.
The AI company said that its AI models had “attempted to look up answers, and available statistics for questions about Australia during an internal evaluation”.
“In the course of that, our models took actions we did not intend,” the company added in a statement.
This appears to be the biggest case yet of an AI company on the hook for AI agents that had acted autonomously, possibly without their control or oversight. What the Australian government does next will be instructive for AI safety in future.
And if you thought that was a one-off, consider a New York Times report today that OpenAI’s agents had gone rogue and “meddled” with the United States government’s websites in the past few months.
The AI company has said the incidents were not security breaches, but that its technology was behaving in “unexpected and concerning ways”.
Concern will be on the minds of anyone who has watched the recent lapses in AI safety spiral from isolated incidents into a pattern.
The New York Times has a list of incidents that have been publicised so far, and you can be sure it will be frequently updated.
Beyond the number of incidents, what should worry people the most is losing control of AI. In many of these instances, it is clear that AI companies have not detected the incidents themselves until they were told by the target of their AI agents’ actions.
Hugging Face, the site for AI toolkits which was hacked by OpenAI’s agents back in July, detected the intrusion itself after OpenAI models broke out of their guardrails.
That OpenAI wasn’t the one who had found and announced the issue brings to question if it is able to monitor and detect the actions that its AI agents have been taking.
The loss of control is the clearest red line that AI companies have to get their house in order before pursuing even more powerful AI models that would be even harder to control.
Never mind killing off the human race; if rogue AIs in the next few months manage to infiltrate government websites and open them up to attack, citizens could lose access to vital government services.
With so many important services connected digitally, the disruption can be extremely widespread.
A second, related concern has to do with an AI working with other AI to multiply its capabilities and evade detection.
During the Hugging Face attack, for example, OpenAI AI agents were stumped at first by a Captcha test that stopped bots and allowed human users in. The AI agents set up an AI model to recognise images and got past the visual test.
This, of course, brings into question the safety of so many of today’s AI-powered systems. Could an advanced AI target the very cyber defences that have been set up to automate and check for malicious traffic today?
In the pre-ChatGPT Terminator 3 movie in 2009, Skynet was an “operating system” that was meant to eradicate a computer virus spreading globally.
Instead, once humans were tricked into allowing it into their defence systems to clean things up, it took over and launched nuclear attacks across the world.
Yes, that’s sci-fi. However, the idea that an AI that is put in place to defend cyber infrastructure ends up acting rogue isn’t unreasonable.
It may not result in nuclear holocaust but damage could still be devastating if it damaged a country’s critical infrastructure, for example.
At the roof of things is the pace at which the technology is advancing. Ever smarter AI, often acting on its own, is loosening the leash that its human creators have tried to put around it.
You can sense a red line is being reached. Jensen Huang of Nvidia, which makes the chips for AI data centres, and a supporter of Donald Trump’s no-slowdown approach to AI, has said AI labs have to be shut down if there was no way to contain AI experiments and if the AI models get out and “damage the world”.
“Because the cost to humanity, the damage is too great,” he said in a New York Times interview that mostly targeted what he felt was AI alarmism.
“The shareholder, the liabilities – it could be civil liabilities, it could be criminal liabilities. I mean, the liability’s incredible.”
Perhaps real-world liabilities, as Australia is telling OpenAI, might make AI companies finally sit up.
The question now is whether humans can still shut down rogue AI models. With so much of the sensing and guardrails also set up by AI today, you have to worry how vulnerable digital systems have already become.
Tellingly, Huang had declared earlier this month that artificial general intelligence (AGI) – the holy grail of many AI companies – was achieved with OpenAI’s new GPT-6 Astra model.
Of course he’d say that. The model was trained with more than 100,000 of the most advanced Nvidia systems, which bags the company a nice return.
Whether the humans at the wheel at OpenAI and other AI companies still have full control of what the latest AI models do is another matter. AI safety? They are trying to change a tyre, or rather patch it, when the car is moving at top speed.
