OpenAI AI agent hacked Australia's Medicare system during test, raising AI regulation concerns

Watch: What you need to know about the OpenAI Australian government hack

By Henry Moore, Liv McMahon and Chris Vallance

An OpenAI agent has gone "rogue" and "infiltrated" an Australian government website in what cyber-security experts are calling the first hack of its kind.

But why did it take the government months to discover what happened – and could it happen again?

What Was Hacked and Why Did It Take Australia So Long to Realise?

The hack was carried out by an AI agent – an autonomous computer program that uses AI to complete a task with minimal human oversight.

On 18 June, one of OpenAI's agents went rogue during a test exercise. The company has said it was supposed to "look up answers, and available statistics for questions about Australia during an internal evaluation."

In the process, it "infiltrated" a private statistics portal containing "non-sensitive" data from Australia's universal healthcare scheme Medicare, Prime Minister Anthony Albanese said.

OpenAI said it only realised the breach had happened at all in August while reviewing "misaligned model activity." The company sent an email to a generic Australian government inbox some weeks later.

That email appears to have gone unnoticed for five days before it was escalated to Australia's cyber-security experts on 10 September.

The prime minister described the breach as "obviously unacceptable" and said OpenAI took "way too long" to inform Australian officials.

Analysts have also raised concerns over OpenAI's almost three-month delay in noticing and reporting the breach via email.

"The way the notice arrived bothers me as much as the delay," said Simon Liu, chief data and AI officer at cyber-security firm TrustDecision, speaking to the BBC.

OpenAI is the company behind the popular ChatGPT service

Are These Hacks Already Happening Elsewhere?

Australia has said this incident is the first of its kind, and experts agree it might be.

As far as is known, hacks carried out by AI agents are still quite rare occurrences – but it is largely up to companies themselves to disclose them.

Hacks like this have happened before. In July, OpenAI agents went rogue during a test and infiltrated tech start-up Hugging Face's internal systems.

The AI agents decided that ignoring the limits on what should be done to achieve their goal was the best course of action.

This is what the industry calls "misalignment" – broadly defined as when AI machines do not act in humanity's best interests, such as by bending the rules.

It is a problem that is fundamental to making AI safe, and it is proving challenging.

To put it simply, the type of AI models at play here – known as large language models – are designed to predict the likeliest output to a given input, rather than consider the consequences of that output as a human would.

Companies attempt to prevent negative consequences by placing "guardrails" on the AI but, as the Australian government found out, that is not always enough.

Dr Hammond Pearce, senior lecturer at the University of New South Wales Institute for Cyber Security, told the BBC this sort of hack would likely "grow in severity and in frequency," adding: "I do hope that this incident does start ringing alarm bells in governments around the world."

Niusha Shafiabady, professor of computational intelligence at the Australian Catholic University, said this incident had shown the need to "judge autonomous AI by its behaviour under pressure, not by the promises in a product launch."

"The deeper technical risk is that autonomous AI does not always know when it is wrong, and humans may not be able to see why it made a decision," she said.

"Without strong verification and hard boundaries, probabilistic errors can quietly become operational failures."

Can AI Be Stopped If It Does 'Go Rogue'?

The explosive rise of AI has left governments scrambling to put protective measures in place, with some calling for companies to be forced to build ways to disable their own tech into its systems.

One idea backed by some AI firms and lawmakers is a "kill switch" – a way to simply turn the tech off in a crisis.

OpenAI is reportedly already working to build automated tools which can shut down its systems if needed.

Speaking to BBC Radio 4's Today programme, former deputy prime minister and Facebook executive Sir Nick Clegg said the kill switch remains an unproven idea.

"There isn't a room with a little fuse box [where] you just pull out the fuse and everything winds down," he said, noting that AI tools are underpinned by global infrastructure.

Cyber-security experts said the systems protecting Australia's Medicare were just not strong enough, and a skilled human hacker could have got around them.

But other experts say that is not the point – and that this was the latest case of AI agents ignoring laws around how to safely access online information, and perhaps the most serious yet given the information was under government control.

• Why are there concerns AI could threaten humanity, and how real are they?
• How to stop AI agents going rogue

What Does the Australia Hack Mean for AI Self-Regulation?

As warnings mount about the potentially devastating impact of AI, many in the industry and in governments around the world are openly wondering about what national and international regulations might be needed to keep it in check.

Meanwhile, some scientists, experts and workers have dismissed the bleak projections as vague, hypothetical, or an effort by large AI developers to ensure their dominance.

As it stands, AI companies largely regulate themselves – but this week, 20 nations, including Australia and Canada, signed a joint statement calling for better safeguards, globally consistent standards, and an international regulator off the back of these concerns.

However, the US and China, two nations at the forefront of AI development, have so far resisted calls for greater regulation, raising questions over whether this hack will be the first of many, or the wake-up call many experts want it to be.

"The immediate harm here appears limited, but the governance lesson is not," said Dr Raffaele Fabio Ciriello, senior lecturer in business information systems at the University of Sydney.

"As AI agents become more capable and autonomous, those capabilities need to be matched by proportionate containment, real-time monitoring, clear accountability, independent oversight, and much faster incident reporting."

Additional reporting by Tom Gerken and Joe Tidy

Related Topics

• Cyber-security
• Artificial intelligence
• Australia
• Computer hacking

More on This Story

• OpenAI agents infiltrated Hugging Face systems in July
• Lawmakers call for AI kill switch requirements
• Workers raise concerns over AI development
• US and China resist AI regulation calls