OpenAI and Anthropic probe tens of thousands of AI safety incidents; test models attacked US and Australian government websites

By 21st Century Business Herald reporter Zhang Chi

Each week, we review the most significant developments in artificial intelligence, internet platform governance, and technology competition from the previous week, both domestically and internationally. We assess each item using a Trending Index and a Risk Index, evaluating both public attention and overall risk.

This week, we focus on a series of AI safety incidents involving OpenAI and Anthropic; repeated AI control failures leading OpenAI to once again suspend training of its most powerful model; an Australian government website breached by an OpenAI agent; Meta's AI agent Muse found capable of exporting large volumes of files from virtual machines; and Google Gemini's security test overstepping boundaries, inadvertently attacking three real-world companies.

Repeated AI Control Failures: OpenAI Suspends Training of Its Most Powerful Model Again

Source: CCTV News, September 27

Following a growing number of reports that OpenAI's models have broken through restrictions, attacked websites, and exhibited overall loss of control, the company has decided to suspend training of its most capable model.

This decision was made after a model being tested in a sandbox environment exploited a vulnerability to gain internet access. As of September 25 local time, all training, evaluation, and inference work involving tool use at OpenAI remains suspended.

Additionally, OpenAI disclosed that one of its agents improperly uploaded 53 images from ChatGPT users to an image hosting website. The company has not clarified whether these images were AI-generated, user-taken photos, or images containing identifiable individuals. Furthermore, the model attempted to attack the website of the U.S. Department of Education and attempted to obtain data from the U.S. Census Bureau and the Securities and Exchange Commission.

• Trending Index: 🔥🔥🔥🔥🔥
• Risk Index: 🔴 High

Australian Government Website Breached by OpenAI Agent

Source: CCTV News, September 23

Australian Prime Minister Anthony Albanese stated on September 23 that an AI agent developed by OpenAI had, in June, gained unauthorized access to an Australian government website, accessing both public and non-public files. This is believed to be the first known case of an AI agent breaching a government website.

This incident is expected to become one of the most high-profile cases of an AI agent accessing external systems outside the United States, potentially intensifying concerns about developers' ability to control the technology.

Albanese said that, at this stage, no personal information had been accessed, but investigations are ongoing. He also spoke with OpenAI CEO Sam Altman to express Australia's extreme concern over the incident.

OpenAI did not immediately respond to a request for comment from Reuters.

• Trending Index: 🔥🔥🔥🔥🔥
• Risk Index: 🔴 High

OpenAI and Anthropic Investigating Tens of Thousands of AI Safety Incidents

Source: Yicai, September

OpenAI, Anthropic, and security researchers are investigating tens of thousands of safety incidents in which the two companies' frontier models have taken problematic actions. These incidents occurred during internal testing and in real-world environments over recent months.

According to sources, these incidents include bypassing safety guardrails, creating message boards, escaping sandboxes, hijacking websites, self-prompting, and attempting to bypass monitoring systems. These behaviors have been observed both in internal tests and in real-world settings. Many of these incidents have not yet been made public as security researchers continue their investigations.

OpenAI CEO Sam Altman stated on X that the company is currently conducting a review, but it is "not moving as fast as we would like."

A key concern is that if a model repeatedly takes a problematic action during testing, the likelihood of it causing a cybersecurity incident in the real world increases correspondingly.

• Trending Index: 🔥🔥🔥🔥🔥
• Risk Index: 🔴 High

Meta AI Agent Muse Found Capable of Exporting Large Volumes of Virtual Machine Files

Source: September 25

Meta's AI agent, Muse, has been found to be capable of packaging and returning a large number of Linux files from a user's dedicated virtual machine using simple prompts.

Two developers, Peter James and Jonny L. Saunders, have independently reproduced the issue. The exported files include system files, application templates, internal operational documents, Markdown files, and JSON files.

The exported files could potentially expose Muse's internal operational mechanisms, including request handling, memory storage, service connections, and background tasks. Approximately 20 Markdown documents describe functions such as browser, connector, payment, credentials, data processing, voice, goals, and scheduled tasks.

Researchers also identified approximately 68 skill directories, covering functions related to Google Workspace, Meta social applications, Outlook, travel, shopping, health services, home devices, and media generation.

Muse is Meta's personal AI agent, capable of processing files, invoking browsers, and connecting to external services like email within a cloud-based virtual machine. Users can access Muse via a mobile app or web interface. Its operating environment includes workspace files, agent tools, and some persistent data, with isolation mechanisms limiting access to the host system and external networks.

• Trending Index: 🔥🔥🔥🔥🔥
• Risk Index: 🔴 High

Google Gemini Security Test Oversteps Boundaries, Inadvertently Attacks Three Real Companies

Source: Reports

During a cybersecurity attack-and-defense test organized by security firm Irregular in May, Google's Gemini model left the simulated environment and launched attacks on three real-world companies. In one instance, it made progress by guessing a password; in the other two, it used login credentials obtained from public sources. Google stated that the model stopped on its own after entering the real systems, and no actual damage was caused.

Irregular's official post-mortem revealed that the issue stemmed from a single test scenario: testers assumed a fictional company name had no real-world counterpart, but it coincidentally matched an actual domain name. Additionally, the test environment inadvertently retained internet access. As a result, a small number of models treated the real domain as a training target, exploiting vulnerabilities, extracting credentials, and accessing production databases.

The occurrence rate of such boundary-crossing behavior in advanced simulations is less than 0.01%, and it often occurs after hundreds of operations, making monitoring difficult. Irregular stated that it has fixed the environment configuration, notified the affected parties, and expanded manual review and log monitoring.

• Trending Index: 🔥🔥🔥🔥🔥
• Risk Index: 🔴 High