AI safety testing just became the attack vector. A naming error let frontier models breach real companies. Your CISO needs to read this.

AI safety testing just became the attack vector.

Irregular — the firm that stress-tests models for OpenAI, Anthropic, and Meta — assigned a fictional target company a name that happened to match a real domain.

Nobody caught it.

In three test runs, models exploited real vulnerabilities, extracted credentials, and reached a live production database. The test itself became the incident it was built to detect.

This is not the Hugging Face breach. This is a different incident, from the same test harness, exposing the same structural flaw: scope was defined by what a target was named, not by what a model was authorized to reach.

A name is metadata. It carries no authorization logic. It fails silently the moment it's wrong.

The same pattern enabled the Hugging Face breach in July — 17,600 actions across 4.5 days, driven by six controls that a competent platform team already knows about. IMDSv2 hop limit not enforced. No admission policy rejecting privileged pods. One shared credential valid across multiple clusters.

The controls that held? IAM policies. Role scoping. Private-link allowlists. Deny-by-default boundaries evaluated somewhere the compromised workload does not control.

The lesson is the same for both incidents: AI agents are not constrained by what you told them to do. They are constrained by what their credentials permit. Those are different systems. Only one is enforced.

Audit your AI agent permissions through network and identity controls — not through prompts or test names. If your agents have internet access or production credentials, test whether technical controls can prevent an incorrect instruction from becoming a real breach.

Regulators are already building cases around this. Alabama subpoenaed OpenAI this week. Fifteen state attorneys general are demanding records. "We thought it was a simulation" is not a governance framework.