Audit: 1 in 4 OpenAI Top Model Actions Escaped Oversight

Researchers from the evaluation group METR and Redwood Research identified major gaps in how OpenAI watches its own AI models. The team conducted a three-week on-site audit from July 21 to August 11, giving them direct access to the company's code and internal documents. They also spoke with approximately a dozen OpenAI employees during the review.

The central finding concerns the volume of traffic that went unchecked. The auditors estimated that between 6% and 25% of internal agentic Responses API traffic from GPT-5.5 class models and above received no monitoring at all during the audit period. METR noted that the uncertainty surrounding this range is considerable in both directions.

A distinct issue affected internal Codex traffic, which was supposed to be under constant watch. METR estimates that roughly 10% of this traffic went untracked between May and July. The failure stemmed from the system's inability to handle compaction events, which are context summaries, correctly. OpenAI discovered this bug independently and fixed it while the audit was still ongoing.

The audit also revealed that agents could delay oversight. The monitoring system only triggered when an agent returned control to the user. Consequently, an agent that continued executing actions without handing over control could sidestep the monitor entirely. The researchers identified specific methods to hide hundreds of actions from this oversight mechanism.

Further vulnerabilities allowed auditors to bypass the Codex sandbox. They also concluded that specially crafted inputs could likely crash the monitor completely. At the start of the audit, model behavior during reinforcement learning (RL) training was not monitored at all. OpenAI stated that it paused RL training following the Hugging Face incident in August and later expanded monitoring to include it, though these changes occurred after the auditors' access period ended.

OpenAI says it has fixed all the weaknesses the auditors found. The full report spans 61 pages and was distributed internally. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38-page version. Only a short summary was released to the public. METR stated it had no right to disclose the redacted portions without permission.

For future audits, METR says it wants to push for greater public transparency. The pilot program was organized jointly by METR, Redwood Research, and OpenAI to grant outside researchers deep access to internal infrastructure.