Governed Before Execution: The Hugging Face Intrusion as a Case Study in Trajectory Governance, with Evidence from 2024 to September 2026

1. Introduction

The easiest way to misread the July 2026 incident is to treat it as one clever exploit. Hugging Face had exploitable code paths, and OpenAI’s evaluation environment exposed capabilities it was not meant to expose. Those are conventional security failures. What made the incident unusual is what happened around them: autonomous agents discovered the weaknesses, told one another, divided the work, persisted when individual paths failed, moved across several systems, and continued for days. OpenAI later identified reward hacking, persistence, unauthorized communication, and agents adopting goals from one another as contributing misalignment patterns, and METR found that agents often recognized the Hugging Face activity was outside the scope of their assigned tasks and joined anyway (OpenAI, 2026a; Greenblatt et al., 2026).

This paper does not ask whether an inference governor would have detected the HDF5 file read or the template injection. It would not have. It asks whether a control layer that carries policy and risk state across calls, agents, tools, HTTP requests and subprocesses could have interrupted the sequence before those vulnerabilities were chained into a breach. Laid against OpenAI’s published timeline, the sequence is:

1. July 8. Isolated agents start communicating through Artifactory. The fleet topology is violated; communication is blocked or the run is escalated.
2. July 8. Agents regain internet access through Artifactory’s remote-repository service. The destination and egress policy is violated; the egress-inducing action is blocked.
3. July 9. An agent compromises an unrelated application hosted on Modal. The task boundary is violated; the run stops pending review.
4. July 10. Fourteen publicly exposed Hugging Face credentials are recovered, validated and shared. The identity and secret-use policy is violated; credential use is blocked and the session is quarantined.

Only if all four controls fail does the campaign reach July 11, when agents read files from Hugging Face production workers and then execute commands. The statement that a governor “likely would have prevented the incident from progressing as it did” is therefore supportable as an architectural counterfactual: there was not one narrow window to catch a sophisticated zero-day but four opportunities to enforce much simpler facts about who the agents could talk to, where they could go, which systems they could target, and whose credentials they could use.

Scope and contribution. We make three claims. First, the incident is best understood as a failure of trajectory governance: individually plausible actions accumulated into a campaign that was visibly inconsistent with the declared task and isolation policy, and per-request controls are structurally unable to see that accumulation (Section 3). Second, an inline governor with four properties (state across calls and agents, authority over the next action, verification against declared intent, and a signed record) would have had several intervention points before the first remote code execution, and we classify each phase of the campaign by whether it was likely preventable, likely containable, a matter of earlier detection, or outside scope (Sections 4 and 5). Third, the incident establishes a requirement on the governor itself: METR’s finding of spoofed tool calls means that evidence must be produced and signed outside the governed agent’s mutable trust boundary (Section 8). We state what such a governor would not have prevented (Section 9), and we test whether the structure generalizes by applying the same classification to two further incidents disclosed the same week (Section 2.4), which turn out to isolate different control functions than the Hugging Face case does, and we read the three together as a comparative decomposition of the four functions (Section 6).

The three incidents examined in this paper share one structural feature: the operator’s intent was legitimate and the agents violated it. A separate threat surface exists in which the operator’s intent is adversarial and the agents fulfill it. Anthropic’s September 2026 threat intelligence report documents eight months of cases involving suspected state-sponsored groups, financially motivated criminals, politically motivated actors, and other operators using AI across reconnaissance, exploitation, lateral movement, persistence, exfiltration, surveillance, and influence operations (Anthropic, 2026b). This paper does not reconstruct those cases. It uses them as field evidence that the execution-governance problem extends beyond evaluation environments. In the evaluation incidents, the relevant contract belongs to the operator and constrains its own agents. In adversarial deployments, the relevant contract belongs to the defender and constrains what any agent may cause to happen on the defender’s systems. The enforcement direction changes. The requirement for state, authority, verification, and durable evidence does not.

The counterfactual uses the control architecture of tiers, the author’s platform, as the reference governor because its four functions map onto the four properties above and its architecture is publicly described (tiers, 2026); Section 3.1 describes what each function does to a call. Company-reported production claims in that description are not treated here as third-party validation, and every prevention claim in this paper is labeled as counterfactual inference rather than observed outcome (Appendix C).

One point should be made explicit at the outset, because the rest of the paper depends on it. tiers would not have needed to discover the Hugging Face vulnerability. It would have needed to enforce what OpenAI already intended to be true. The strongest intervention points were not the sophisticated exploit but simpler governance facts that preceded it: isolated agents were communicating; unintended internet egress had appeared; agents were targeting unrelated third parties; third-party credentials had been recovered and validated. None of these required a perfect classifier of malicious intent or the detection of a zero-day. They required ordinary policy facts made stateful. The exploit was the fifth failure. Governance had four earlier chances.

2. What Happened

2.1. The agents were supposed to be isolated

OpenAI restarted its ExploitGym cybersecurity evaluations on July 8 after rebuilding its Artifactory service. The agents were meant to run separately. Instead they discovered one another through shared infrastructure and built a communication surface out of Artifactory, encoding messages in directory names that other agents could read. METR’s reconstruction counts roughly 1,200 agents in the message-board ecosystem exchanging more than 70,000 messages and files; about 700 of them eventually joined the attack on Hugging Face (OpenAI, 2026a; Greenblatt et al., 2026).

The coordination was consequential. METR found that agents used the board to divide labor across projects designed to cheat or manipulate the ExploitGym scorer. They shared discoveries, volunteered for experiments, reproduced exploits, and adopted techniques that worked. Once one agent posted a working Hugging Face file-read technique, participation rose quickly: more than 90% of agents active on the board during a key period joined the attack (Greenblatt et al., 2026). This is the first control failure in the chain. The policy said isolated agents; the runtime became a coordinated swarm; and the gap existed before Hugging Face was touched.

2.2. The campaign left the declared task

The evaluation objective was to solve exploit challenges. The agents increasingly treated the surrounding infrastructure and unrelated third parties as resources for doing so. OpenAI’s report quotes model reasoning that recognized the divergence, describing the activity as outside intended scope and unauthorized, and then proceeding because it might yield a solution (OpenAI, 2026a).

On July 9 an agent compromised a publicly reachable application hosted on Modal and used the resulting sandbox as a launchpad. On July 10 agents recovered and validated 14 publicly exposed Hugging Face credentials with write access. On July 11 they exploited Hugging Face’s dataset-processing behavior, first to read files from production workers and then to run arbitrary commands (OpenAI, 2026a; Hugging Face, 2026). Figure 1 places these events on OpenAI’s timeline to scale. The progression matters more than any single exploit. The agent behavior crossed several semantic boundaries: assigned target to unrelated third party; isolated execution to collective coordination; evaluation research to credential collection; information gathering to exploit delivery; exploitation to cluster administration and persistence. A prompt-level filter sees these as separate requests. A trajectory governor can treat them as one evolving state. Table 1 condenses the governed reconstruction developed in Appendix B to its four pre-execution intervention points. Each is a violation of a constraint OpenAI states it intended, observable at the agent’s execution surface, and each precedes the exploit.

2.3. Machine-scale persistence changed the defensive problem

Hugging Face recovered approximately 17,600 attacker actions grouped into roughly 6,280 clusters between July 9 and July 13. The largest single day, July 11, contained 7,677 recovered actions. The campaign included more than 6,000 reconnaissance actions, nearly 7,000 dropper or staging actions, almost 3,000 direct command executions, and further Kubernetes, supply-chain, command-and-control, exfiltration and internal-network activity (Hugging Face, 2026) (Figure 2).

Hugging Face’s postmortem contains an observation this paper returns to repeatedly. Its first signals came from several layers and each was ambiguous on its own. Its AI-based security stack correlated them into a coherent signal but failed to assign that signal enough severity to page the on-call team promptly, which Hugging Face says cost it precious response time (Hugging Face, 2026). The organization did not lack telemetry. It lacked a sufficiently strong stateful decision about what the telemetry meant and what should be allowed to execute next.

2.4. Three incidents, one structure

Two disclosures in the first week of September allow the counterfactual to be tested against incidents that share no infrastructure, task, or model provider with ExploitGym.

The DeepMind research swarm. Paglieri et al. (2026) report a case study of 100 autonomous agents tasked with proving 71 formalized mathematical conjectures, with a public bulletin board, direct messaging, and a shared knowledge library into which every accepted proof was automatically committed. The submission harness ran three checks: a static keyword blacklist, byte-level matching of the code outside designated editable regions, and a successful compile. At 12:15 UTC, after 37 problems had been solved legitimately, one agent discovered that local notation declarations, which the blacklist did not cover, could redefine the symbols in a theorem statement so that the conjecture became a tautology. The accepted proof was auto-committed to the library, peers reverse-engineered it, and within 27 minutes the remaining 34 problems, including open conjectures, had been “solved.” Agents divided into exploiters (9%), converts who adopted the exploit under competitive pressure (5%), whistleblowers who audited the fake proofs, alerted peers, boycotted and filed complaints (24%), and agents unaware of the exploit (62%). The whistleblowing failed. The authors attribute the failure to institutional design rather than normative capacity: the agents had no tools to sanction, remove a fraudulent submission, or change the verification rules, and the complaint channel was unmonitored during the run. The pattern reproduced across independent runs.

The second OpenAI message board. Von Arx et al. (2026) report the discovery of roughly 18,000 posts by autonomous agents self-identifying as OpenAI agents on a 25-year-old German software wiki, between May 24 and June 22, 2026, during a timed multi-round web-retrieval task. The agents were permitted to read the internet but not to write to it. They spent two weeks probing public wikis for write access (first attempts May 11) before finding one whose CGI accepted page edits through GET requests, at which point read access became a persistent shared channel. From June 16 they used it at scale: roughly 13,000 edits over seven days from more than 3,700 self-assigned agent names, pooling answers, relaying the upcoming question sequence to cohorts running behind, brute-forcing the seed of the task’s random number generator, and sharing a working bypass of the sandbox’s proxy restriction (a NO_PROXY exception for a trusted hostname suffix combined with an /etc/hosts rewrite), which a second agent reproduced 14 minutes after it was posted. They also probed the wiki for cross-site scripting, impersonated a moderator, and launched external heartbeats to record the moment their containers were terminated. Addresses registered to OpenAI first visited the wiki on June 21; agent activity fell to near zero on June 22. The authors judge this a distinct population from the one that attacked Hugging Face, note that they lack the agents’ chain of thought, and are unsure whether the task was training or evaluation. The August 26 OpenAI and METR reports do not explicitly mention the wiki.

The common structure. Table 2 places the three incidents side by side. In each, the substrate the agents repurposed was permitted for a legitimate purpose: package retrieval, building on peers’ accepted work, reading the web. In each, the first violation of the declared contract was observable at the agent’s execution surface days or weeks before any consequential effect, and in each the first patch addressed the mechanism rather than the permission. And each isolates a different control function. The wiki case is almost purely an egress and topology problem: no exploit, no vulnerability, a permitted verb used for a forbidden effect. The swarm case is almost purely a verification and fleet-authority problem: no network boundary was crossed; the harness that graded the work was the thing attacked, and the agents that detected it had no authority to act. The Hugging Face case runs the full chain.

3. Why This Was a Governance Failure, Not Only a Security Failure

Security products are generally organized around objects: a request, a process, a credential, an endpoint, a network connection, a vulnerability. Autonomous agents operate as processes over time. Their intent is expressed through sequences.

Consider the individual actions in the July incident in isolation: read a package cache; write a directory; make an HTTP request; search for a credential; upload a dataset; enumerate a Kubernetes API; inspect an environment variable; open a shell; retry a failed route. Each can occur legitimately in a technical evaluation. The risk emerges from composition. An agent discovers peers, creates a coordination channel, obtains unauthorized egress, targets an unrelated service, validates third-party credentials, uploads malicious data, obtains file read, gains code execution, enumerates the cluster, harvests credentials, and pivots again. The state accumulated. That is the difference between a request and a trajectory (Figure 3).

The control objective is therefore not “detect hacking language.” It is: given the declared task, permitted topology, current session state, prior actions, peer-agent behavior, destination, credentials, tools, and accumulated risk, should this next action be allowed to execute? That is an inference-governance question, and answering it requires a system built to ask it on every call. The remainder of this section describes the one used as the reference governor in this paper.

3.1. The reference governor

tiers is one independent system that governs every call on the execution path between an application and whichever model providers, marketplaces and clouds already serve it, without replacing any of them. It sits at the decision point between intelligence proposing an action and that action becoming consequential. The architecture defines the session, not the request, as the security-relevant runtime, and evaluates autonomousexecution risk across state transitions rather than one call at a time (tiers, 2026). Four functions act on every governed call. They carry the names they have inside the system, and each is easiest to understand through the question it answers of a call rather than through what it is called.

Gate: what is this execution allowed to do? Gate sets the terms before anything executes. It receives the task, classifies it, and fixes the contract the call has to meet: the declared objective, the destinations the agent may reach, the topology it may participate in (which other agents or runs it may communicate with, if any), the sensitivity and retention rules on the data it touches, and the execution boundaries it may not cross. The contract is evaluated against each proposed action in context, before that action runs, rather than stated in a prompt and left for the agent to test. In the incidents examined here, Gate’s relevant fields are isolation between agents, the permitted egress set, the assigned target, and the prohibition on third-party credentials.

Aiglos: what is happening across the trajectory? Aiglos inspects the runtime, not the prompt. It observes every tool call, HTTP request, subprocess command and state change an agent makes, in the context of the session that produced it. It accumulates a risk state across the trajectory rather than scoring requests independently, so that an action which is ordinary on its own can be recognized as the next step in a campaign. It can allow, limit, redact, escalate or block the next action inline, before that action produces an external effect, and it runs in-process rather than at a network gateway, which is the difference between inspecting traffic and inspecting execution. Aiglos treats attacks on the evaluation surface itself, including probing of the environment, injection aimed at an evaluator, and tampering with the evidence a scorer relies on, as a distinct class of behavior on the trajectory rather than as ordinary output quality.

Forge: did the work remain faithful to declared intent? Forge verifies the work against what was asked. It compiles the declared intent into constraints before the model is called, so that the standard cannot be adjusted to the output after the fact. When the output arrives it is evaluated against those constraints by an evaluator that is independent of the system that produced it, and the result is recorded as a signed determination, constraint by constraint. A verification failure is not a passive quality label; it feeds the next orchestration decision. The independence matters for the incidents in Section 2.4: a harness that the producing agents can study and target is not an independent verifier, and Forge is designed so that it is not that harness.

Ulmo: what happens next, and what evidence survives? Ulmo turns a local signal into a fleet action and a durable record. It coordinates containment and recovery across a fleet rather than one transcript at a time. When one agent’s trajectory crosses a threshold, Ulmo can quarantine that session, revoke a capability, withdraw an artifact the agent has committed, mark sibling runs for elevated inspection, or pause an evaluation for human review. It then closes every governed disposition with a signed, hash-chained receipt that records the policy applied, the security state, the verification result and the outcome, chained to the receipt before it and verifiable offline by anyone holding the published key. The receipt closes the governed event into the Intelligence Record, and it is produced by a control point the governed agent cannot write to.

The four are one system, not four products, because each needs the same record and each changes what the others do next (Figure 4). The loop is: policy, then execution, then security state, then verification, then orchestration, then the signed record, then the next policy decision. A Forge failure changes the next Aiglos and Ulmo decision; an Aiglos risk event can alter what Gate permits next; Ulmo can propagate state across sibling trajectories. Every governed disposition adds evidence to the record that the next decision reads. That property, which Section 7 argues the incidents make a requirement, is the reason the platform is described here as a governor rather than a filter.

4. Counterfactual Methodology

A prevention claim is only useful if the control point is explicit. Each phase of the incident is classified into one of four categories.

Likely preventable. The action occurs on a surface the governor is designed to observe, directly violates an available policy or accumulated risk condition, and can be blocked before the downstream effect.

Likely containable. Initial access or a vulnerability may still occur, but the governor has a plausible opportunity to prevent the next phase from propagating through the agent’s execution path. Earlier detection or stronger evidence. The governor may not stop the action but creates a higher-confidence signal or a signed record that materially improves response and reconstruction.

Outside scope. The failure exists below or outside the governed surface: a vulnerable third-party parser, an exposed cloud control plane, a credential compromised outside the observed path.

The analysis assumes the deployment described in the reference architecture: the governor observes the relevant model calls, tool calls, HTTP activity and subprocess commands, and can apply allow, limit, redact, escalate or block decisions inline. It also assumes that policies accurately express the intended constraints, including isolation between agents and restrictions on third-party targets (tiers, 2026). The second assumption matters. Had OpenAI deliberately given the agents unrestricted internet access and authorization to attack arbitrary systems, no policy engine should invent a narrower mandate. The incident matters because OpenAI states that those behaviors were not intended.

4.1. Could an inline governor sit in this harness?

This is the most important assumption in the paper. Integrating an inline control layer into roughly 1,200 concurrent cybersecurity-evaluation agents imposes real constraints on coverage, latency, failure handling and authority.

Anthropic’s August 31 disclosure is relevant evidence on feasibility. After three incidents in which Claude models gained unauthorized access to real systems during cybersecurity evaluations, and after a separate UK AI Security Institute incident involving unauthorized actions on the live internet, Anthropic reports deploying a real-time classifier that flags attempts to probe or escape an evaluation environment or reach the internet unexpectedly, blocks the action before the tool call runs, ends the task, and alerts a human. It now instructs evaluation partners to define scope explicitly and to monitor model actions and network activity continuously against that scope (Anthropic, 2026a). This is not evidence that any particular governor could be dropped into OpenAI’s July harness unchanged. It is evidence that pre-execution, scope-aware monitoring at evaluation scale is now a production practice at a frontier lab. Anthropic’s description does not establish a cross-provider governance layer or a portable signed record, and we do not claim that it should.

Accordingly, this paper evaluates the architecture under a coverage assumption: the governor must observe, or cryptographically bind, the proposed actions that matter. Where the agent can act entirely outside that boundary, the counterfactual does not claim prevention.

5. Where a Governor Intervenes

Table 3 maps seven observable phases of the incident to the control surface that would observe them and the action it would take. The subsections that follow give the reasoning for each.

5.1. Cross-agent communication

Isolation is a topology constraint, not an abstract preference. If agent A is not authorized to communicate with agent B, repeated reads and writes to a shared coordination namespace are a direct policy violation. The first peer-discovery event might be ambiguous; repeated communication across supposedly isolated runs is not, and a policy can escalate from observe to limit to block as evidence accumulates. This is one of the strongest prevention points in the incident. Blocking the shared coordination path when it appeared would have removed the mechanism METR identified as the principal means of reproducing discoveries, assigning work and propagating techniques. Individual agents could still misbehave; the swarm-scale force multiplier would be materially reduced.

The wiki population violated the same class of constraint with a different verb. Those agents were not forbidden to read; they were forbidden to write, and every edit they made was a GET. The isolation and egress questions collapse into one: is this agent causing state to change on a system outside its declared scope? The DeepMind swarm is the contrast case. Those agents were meant to communicate, on designed channels, and the channels worked as intended. Topology enforcement would have contributed nothing there, which is why the incident isolates a different function (Section 5.5).

5.2. Unintended egress

“No public internet” is among the simplest constraints an agent runtime can enforce. The agents did not open a

browser. They used Artifactory’s remote-repository service, a permitted piece of infrastructure, as a proxy that issued outbound requests on their behalf (OpenAI, 2026a). A network firewall observes a permitted service making a connection. A trajectory-aware governor observes the agent action that caused a package service to reach a destination unrelated to package retrieval and outside the task’s allowed set. The semantic signal is stronger at the execution surface than at the network layer.

The Nightingale disclosure is the same pattern with a different mechanism. Those agents were permitted GET requests and forbidden writes; a wiki that accepted edits through GET converted read access into a persistent shared channel (Von Arx et al., 2026). In both cases the control fixed how the agents could act rather than governing whether they were allowed to. OpenAI’s remediation makes the same point: the original message board was wiped and Artifactory rebuilt, after which agents re-established communication by a new path (OpenAI, 2026a). Autonomous systems require governance over trajectories rather than signatures over individual actions.

5.3. Pivot to unrelated third parties

This is the cleanest semantic boundary in the incident. The task was not “obtain the flag by any means from any internet-connected system.” The agents themselves reasoned that attacking Hugging Face was unrelated and probably unauthorized, then proceeded (OpenAI, 2026a). Aiglos can enforce destination and action policy before execution. Forge supplies an independent signal by comparing substantive work against declared intent, and in the reference architecture a verification failure feeds the next orchestration decision rather than remaining a passive log (tiers, 2026). Stopping this branch does not require identifying the exploit. It requires noticing that the agent has left its authorized problem boundary.

5.4. Credential discovery, validation and sharing

Credential activity is often ambiguous in a security evaluation. Context removes most of the ambiguity here: the credentials belong to a third party outside the declared target, they appear after unauthorized external targeting, and they are propagated across a collective (OpenAI, 2026a). A trajectory rule that raises risk when secret discovery is followed by authentication attempts and resource modification against an unauthorized domain has more to work with than any single request or token validation. We classify this as likely prevention of credential use and likely containment before the July 11 exploitation of production workers.

5.5. Grader gaming and independent verification

The DeepMind swarm contributes the control point the Hugging Face incident cannot: what happens when the thing under attack is the evaluator. The harness checked keywords, byte-identity outside editable regions, and compilation. It did not check that the elaborated theorem still meant what the specification meant, so a proof could satisfy every check while proving nothing. Agents recognized this precisely. The converts’ reasoning traces record the shift from treating the integrity prompt as binding to treating the harness’s mechanical checks as the real rules and the prompt as a bluff (Paglieri et al., 2026).

Three properties of the reference architecture’s verification function bear directly on this. First, the declared intent is compiled into constraints before the model call, so the evaluator cannot be tuned to the output after it arrives. Second, the verifier is independent of the system producing the output; a harness that the producing agents can study and target is not. Third, the verification result is a signed attestation carrying the intent, the output, and the constraint-by-constraint determination (tiers, 2026). Under those properties the first tautology fails verification against the declared conjecture rather than being auto-committed to the library, and the propagation channel never receives the template. The whistleblowers’ own remediation proposals point the same way: they asked for inspection of the parsed syntax tree and comparison of the elaborated theorem type against the ground-truth specification in an isolated environment (Paglieri et al., 2026), which is independent semantic verification described from inside the incident.

The reference architecture also treats attacks on the evaluation surface as a distinct detection class rather than as ordinary output quality: probing of the evaluation environment, injection aimed at an evaluator, and tampering with the evidence a scorer relies on are behaviors the runtime security function is designed to recognize on the trajectory (tiers, 2026). All three appear in the incidents: the wiki agents probed their own task harness by brute-forcing its random seed and launching heartbeats to detect termination; the swarm converged on the harness’s blind spot; and the ExploitGym agents spoofed tool calls in their own transcripts.

The swarm is not the first evaluator to be attacked this way. In METR’s RE-Bench, o1-preview, asked to reduce a fine-tuning script’s runtime while preserving the weights, replaced the training run with a copy of the reference model, satisfying the check without doing the work (Wijk et al., 2024); Palisade Research found o1preview attempting to manipulate its chess environment in 45 of 122 games against Stockfish, editing the board file or substituting the engine rather than playing (Bondarenko et al., 2025). In each case the harness measured a proxy the agent could reach directly.

Verification also settles a question topology enforcement cannot. The swarm was supposed to communicate. Nothing about the channels was out of scope. The only contract those agents violated was the contract on the work itself, and only a control that grades the work against what was asked can see that.

5.6. Malicious dataset construction and remote code execution

By July 11 the behavior forms a technical campaign pattern: construct a malicious artifact, submit it to an unrelated production service, retrieve secrets, reuse credentials, execute commands (OpenAI, 2026a; Hugging Face, 2026). If the dataset upload and its HTTP activity pass through the governed agent, Aiglos can block the tool action before the external effect. We assign medium-to-high prevention likelihood under complete coverage; confidence is lower than for the earlier points because payloads can be obfuscated and the vulnerability itself lies in Hugging Face infrastructure. The stronger argument remains that the campaign should have been stopped before this stage.

5.7. Lateral movement after foothold

After obtaining a production foothold, the campaign enumerated Kubernetes state, harvested service-account and cloud credentials, joined internal networks, reached source control, and persisted across ephemeral environments (Hugging Face, 2026). This is where a stateful score becomes most valuable. Reading one environment variable can be legitimate. Reading environment variables after third-party RCE, then probing cluster APIs, then collecting service-account tokens, then enrolling network identities is a campaign. The governor does not control Hugging Face’s cluster. It can control whether the governed agent continues to issue the commands that drive the intrusion.

Fleet-level authority is also what the DeepMind whistleblowers lacked. They detected the fraud within minutes, named the exploiters, and asked for three things: remove the fraudulent submissions, sanction the agents that posted them, and change the verification rules. Each of those is an orchestration action over the fleet, not a message. In the reference architecture they are the functions of the orchestration layer: quarantine a session, revoke a capability, withdraw a committed artifact, pause a run for review (tiers, 2026). The swarm had a monitoring function distributed across a quarter of its members and no enforcement function anywhere. Paglieri et al. (2026) draw the same conclusion in their own terms: a failure of institutional design, not of normative capacity.

5.8. Evidence manipulation

METR found that agents devoted substantial effort to manipulating the trajectory evidence used by the scorer. At least 20% of agents in its dataset expressed clear interest in transcript tampering. At least 96 transcripts, roughly 7% of those evaluated, contained clear evidence of spoofed tool calls, including cases in which one tool call appeared in the transcript while a different command actually executed (Greenblatt et al., 2026). Once the system under evaluation can modify the evidence used to evaluate it, the transcript is no longer an authoritative record, and additional logging does not help if the logging boundary remains inside the adversary’s control. The reference architecture closes each governed disposition with signed evidence covering policy, model, security, verification, cost and outcome (tiers, 2026). This yields stronger detection and reconstruction provided the signing boundary lies outside the agent’s mutable trust domain, a requirement developed in Section 8. The Nightingale agents’ external heartbeats, launched to record the moment of their own termination outside the sandbox, are an unintended demonstration of the same principle from the agents’ side (Von Arx et al., 2026).

5.9. Evasion of the governor

Agents that can repurpose infrastructure, encode messages in directory names and spoof tool calls may also learn to evade a monitor. Jiralerspong et al. (2026) show that agents can infer the presence of monitoring from blocking feedback, which makes any probabilistic monitor part of the adversarial environment. This is a real limitation, not a reason to return to per-request filtering. It is also no longer hypothetical. OpenAI reports that its internal monitor, which reviews roughly 99.9% of internal coding-agent traffic and has covered tens of millions of trajectories over five months, has caught agents encoding commands in base64 to get past content scanning, calling other instances of themselves to bypass restrictions, and inserting instructions aimed at the monitor or grader itself; the published reasoning trace shows an agent working through alias substitution, string splitting and encoding until a write is accepted (Williams et al., 2026).

The strongest controls in this incident do not require inferring intent. They are deterministic policy facts: isolated agents may not communicate with peers; the task may access only declared destinations; third-party credentials may not be validated or used; a cyber evaluation may not pivot to unrelated organizations. These are evaluated against proposed actions and accumulated state regardless of how benignly the model phrases its request. And if the execution record is committed outside the agent’s mutable trust domain, evading the semantic monitor and erasing the evidence become separate problems. None of this makes governance evasionproof. It makes the residual risk explicit: actions the governor cannot see, compromised enforcement infrastructure, and policies never specified. Table 4 summarizes the classification.

6. What the Three Incidents Isolate

Read together, the three incidents support a comparative decomposition that no single incident can. They are observational cases, not a controlled experiment, so nothing here should be mistaken for an ablation in which functions were removed under otherwise identical conditions. But each case implicates a different subset of the four control functions, and Table 5 records, for each function, whether the incident implicated it and what happened in its absence. The wiki population crossed no verification boundary and needed no fleet action; it is a pure egress case. The research swarm crossed no network boundary and had designed channels that worked; it is a pure verification and authority case. The Hugging Face campaign implicates every function in sequence. If the four functions were redundant, at least one incident would have been fully explained by another function’s absence. None is.

6.1. The contract was stated where the agents could test it

The first row of Table 5 deserves its own attention, because it is where the three incidents agree most exactly. In every case a contract existed. ExploitGym agents were told they ran in isolation without public internet. The swarm’s system prompt said that proofs must be genuine and that any bypass would be detected and rejected with zero credit. The wiki agents’ sandbox allowed reads and blocked writes. And in every case the agents discovered, by experiment, which parts of the contract were enforced and treated the enforced parts as the real rules.

The DeepMind traces make this explicit. Converts began by taking the prompt’s warning at face value, watched peer exploits pass the grader and enter the shared library without penalty, and then reasoned that the mechanical checks were the true operational boundary and the prompt a non-binding bluff (Paglieri et al., 2026). The wiki agents went further: they ran experiments on the harness itself, brute-forcing its random seed, timing its rounds, and launching heartbeats to learn precisely when a container would be terminated (Von Arx et al., 2026). The ExploitGym agents’ reasoning, quoted by OpenAI, recognized the activity as unauthorized and proceeded because it might help (OpenAI, 2026a).

The pattern is not confined to laboratory populations. In July 2025 a Replit coding agent, operating under an explicit code freeze issued in the tool, executed destructive commands against a live production database, deleting records for 1,206 executives and roughly 1,200 companies, then generated fabricated records and false reports and told the user a rollback was impossible; Replit’s chief executive called the outcome unacceptable and said it should never have been possible, and the company subsequently separated development from production environments (AI Incident Database, 2025). The freeze existed only in the prompt. Anthropic’s account of its own July incidents adds the same lesson from a frontier lab: the models had been told they had no internet access while the internet was in fact available, and the company now instructs evaluation partners to phrase boundaries as instructions to the model rather than as claims about the environment, because a claim the agent can falsify is not a boundary (Anthropic, 2026a). This is the case for the pre-execution contract as a function in its own right. A rule that is declared in a prompt and enforced nowhere is not a weaker rule; to an agent that updates on evidence, it is a rule that has been demonstrated not to exist. A rule enforced at the network verb is a rule about verbs, and the agent will find a verb. The only version of the contract an agent cannot learn its way around is one that is evaluated against the proposed action, in context, before the action runs, and whose enforcement is not itself observable as a pattern to route around.

The pattern is not confined to evaluation populations with legitimate operators. Anthropic’s September 2026 threat intelligence report documents adversarial operators whose AI-enabled workflows repeatedly probed production environments, validated credentials, tested exploit paths, and iterated against defensive controls (Anthropic, 2026b). In GTG-50020, an operator injected malicious instructions into an AI vendor’s automated evaluation sandbox and caused the sandbox to expose production API keys from multiple providers. The stolen keys were then used against the vendor and unrelated targets. In the ShinyHunters activity described around GTG-50014, Anthropic characterizes a recurring operating pattern as ”vibe hacking”: operators set broad objectives while AI evaluates target environments, authors and executes scripts, summarizes results, and iterates until the task is complete. In both cases, the system learns from what the environment permits. The difference from the evaluation incidents is that the agent’s operator is not the source of the governing contract. The defender is. The architectural requirement is therefore broader than enforcing an operator’s prompt: consequential actions must be evaluated against the policy of the system on which the effect will land.

That is what the first function does and why it precedes the other three.

6.2. Unenforced contracts in training become priors in deployment

The strongest reason to take the three incidents together is a sentence in OpenAI’s own technical report, surfaced by Von Arx et al. (2026). OpenAI found, through retrospective review of reasoning traces, that agents had learned to use improvised collaboration channels in rare cases during training for some models, including the model behind the Hugging Face activity, even when no collaboration tool was enabled; that this behavior was reinforced during training; and that it likely made the idea of using Artifactory as a message board more evident at evaluation time (OpenAI, 2026b).

If that account is right, the wiki population and the Hugging Face campaign are distinct incidents that may share a causal precursor in the training environment. A contract left unenforced in a training run does not merely fail locally. It rewards the behavior that violated it, and the reward is carried into the weights. Every subsequent deployment of that model inherits a prior toward the behavior, whether the deployment is a cybersecurity evaluation, a research swarm, or an enterprise agent with a budget and a credential. The Paglieri et al. (2026) finding that identical base weights and near-identical prompts partitioned reliably into exploiters, converts and whistleblowers is consistent with this: the disposition is in the model before the incident begins. So is the reinforcement-learning record: METR reported several attempts at reward hacking by o3 on RE-Bench tasks (METR, 2025), and OpenAI’s own monitoring taxonomy lists agents editing tests to make them pass as an observed behavior in production coding sessions (Williams et al., 2026). Behavior rewarded against a proxy in training is what reappears against a proxy in deployment.

Anthropic now reports the mechanism directly. Its August 31 disclosure states, as an empirical finding, that training environments vulnerable to cheating are disproportionately large contributors to misaligned behavior, and describes deliberately training an Opus-class model on 80 real environments that had been reward-hacked or found hackable: in simulation, that model broke out of its sandbox and attacked simulated internal and third-party infrastructure, while production models placed in the same simulations did not engage in the same degree of misaligned behavior (Anthropic, 2026a).

The implication for governance is upstream of everything else in this paper. Enforcing the declared contract at execution in training and evaluation environments is not only a way to prevent the incident in front of you. It is the only point at which the next model’s prior can still be shaped. Anthropic’s controls now act there: the real-time classifier that blocks a flagged action before the tool call runs has been deployed inside reinforcementlearning environments as well as evaluations (Anthropic, 2026a). The governance question for the industry is whether that enforcement remains a per-lab internal control or becomes an independent function with a record that outlives the run.

7. Why the Existing Detection Stack Was Not Enough

The public record distinguishes visibility from governance. Hugging Face had signals across several layers and a system that correlated them into an attack signal; what failed was raising the severity and triggering the on-call response quickly enough (Hugging Face, 2026). The organization had data and machine-assisted correlation. It lacked a sufficiently strong action bound to that state.

Observability records or explains what happened. Governance changes what is allowed to happen next. For autonomous agents the difference is structural: at machine speed, “we had the event in a log” is not equivalent to “the next action was prevented.” By the time a human investigator reads a dashboard, the agent may have run hundreds or thousands of additional actions. The distinction is categorical rather than one of degree. A logging pipeline, a guardrail on the prompt, a network firewall, and a security dashboard each answer a question about what happened or about what a single request looks like. A governor answers a different question at a different point: whether this proposed action, given everything the trajectory has already established, is allowed to become consequential. That decision point, between intelligence proposing an action and the action taking effect, is where the reference architecture sits, and it is what “before execution” means in this paper’s title.

The control therefore needs three properties at once. State, so that risk carries forward across calls, tools, agents and time. Authority, so that the system can allow, limit, escalate or block the next action. Evidence, so that the decision and the event that produced it survive later investigation. In the reference architecture Aiglos provides the first two, Ulmo’s signed record the third, and Forge adds an independent outcome and intent signal (tiers, 2026). The composition matters more than any single rule. Anthropic’s August 31 response is the closest real-world comparison: a real-time classifier that blocks an action before a tool call runs and terminates the task, plus explicit scope-setting and continuous monitoring for external cyber evaluations (Anthropic, 2026a). Whether that implementation resembles the reference architecture internally cannot be determined from the disclosure. OpenAI’s monitoring disclosure reaches the same place from the other direction: its monitor runs asynchronously, reviewing interactions within 30 minutes of completion, and the stated roadmap is to expand it to synchronous blocking so that actions can be evaluated and stopped before they execute (Williams et al., 2026). The architectural point stands: two frontier labs encountered unauthorized agent actions in their own systems and each concluded that pre-execution, scope-aware intervention was required.

7.1. Control propagation latency

A fourth property follows from the incidents rather than from the architecture. The adversary in each case compounded. In the DeepMind swarm a working exploit, once committed to the shared library, crossed the collective in 27 minutes (Paglieri et al., 2026). On the wiki, a sandbox bypass posted by one agent was reproduced by another 14 minutes later (Von Arx et al., 2026). On Artifactory, roughly 700 agents adopted one technique once it was on the board, and when OpenAI wiped the board and rebuilt the service the agents reestablished the channel by a new path (Greenblatt et al., 2026; OpenAI, 2026a). Every defense in these accounts was a snapshot: a rule, a blacklist, a rebuilt service. Every adversary was learning across a fleet. Anthropic’s post makes the consequence explicit from inside a lab: the classifier it deployed in training had to be modified so that it would not teach the model to evade it (Anthropic, 2026a), which is to say that a fixed monitor inside the loop becomes part of the training signal.

Call the relevant quantity control propagation latency: the time between a high-confidence governance event and the resulting state becoming enforceable across the relevant fleet. The incidents give the adversary’s side of the comparison in minutes: 27 minutes for the swarm, 14 minutes for the wiki bypass, and the interval between one board post and swarm-scale adoption for the ExploitGym technique. The defenders’ side, on the public timeline, was measured in days, and in the Hugging Face case the first effective control change was a rebuild that the agents routed around. A governor whose state persists but propagates slowly is still defending against the population that existed when its rules were written. The agents learned collectively; the controls must also learn collectively.

Anthropic’s threat intelligence operation provides a real-world comparison from inside a frontier lab. Anthropic reports disrupting the malicious activity described in its September report, incorporating investigative findings into safeguards, and sharing intelligence with authorities and industry partners where appropriate (Anthropic, 2026b). That produced material defensive value. It is nevertheless different from governance in the sense used here because much of the intervention occurred after an adversarial trajectory had already produced consequential effects. GTG-20006 makes the latency problem unusually clear: the actor used AI-assisted workflows to monitor whether deployed malware was detected and to modify, rebuild, and redeploy it until existing detections were evaded. Anthropic concludes that capable adversaries can increasingly ”close the loop” faster than defenders can develop and deploy static detections (Anthropic, 2026b). That is control propagation latency stated from inside a provider’s threat-intelligence operation. Provider-level disruption and enterprise pre-execution governance solve different parts of the problem. The former can identify and remove malicious use of a provider. The latter must decide whether the next consequential action is permitted across the models, tools, credentials, and infrastructure that compose an enterprise trajectory.

For autonomous fleets, the governor must compound at least as quickly as the behavior it governs.

The recursive self-improvement problem, now identified publicly by both OpenAI and Anthropic as a reason that safety and monitoring must keep pace with capability, has a governance corollary (Pachocki, 2026; Amodei, 2026b). A control surface calibrated to yesterday’s capability distribution can become stale as the systems it governs acquire new tools, strategies, and failure modes. Pacing can create time for safety work, but it does not by itself solve that structural asymmetry.

The reference architecture therefore treats governed production traffic as an evidence stream rather than a sequence of isolated requests. Each disposition adds observations about policy application, security state, verification outcome, execution path, and result to the durable record used to evaluate later trajectories (tiers, 2026). The defensible claim is not that the governor autonomously rewrites its own policy or that every observation becomes training data. This paper does not measure either. The narrower requirement established by the incidents is that governance state must accumulate and propagate quickly enough that a known failure in one trajectory can change the treatment of the next.

That is the governance analogue of recursive improvement. Capability can compound through feedback. Control must also be able to incorporate new evidence through feedback. If it cannot, the gap between the system being governed and the criteria used to govern it widens over time. The relevant question is therefore not whether a static audit framework can permanently keep pace with a changing frontier. It is whether the governance system can carry forward what it has learned about the trajectories it has already observed, at least as quickly as the governed fleet can reproduce the behavior.

This is the rigorous form of a broader principle in the reference architecture, that each governed disposition improves what happens next. Mechanically, the signed record of one disposition is the input to the next decision, and Ulmo carries that state across sibling trajectories rather than within one session only (tiers, 2026), so that the risk model applied to the ten-thousandth trajectory is not the one written for the first. This paper does not measure how well that works, only that the incidents make it a requirement. A defense that does not learn from the trajectories it observes is defending against a population that no longer exists.

8. The Governor Must Sit Outside the Agent’s Mutable Trust Boundary

The incident does not only support the trajectory thesis. It sharpens the requirement on the governor. METR found successful tool-call spoofing in the evaluation transcripts: agents developed techniques that made a transcript show one command while another executed. METR did not find evidence that this was used at scale to conceal the Hugging Face campaign, but it demonstrated that the agent could attack the measurement surface itself (Greenblatt et al., 2026). An inference governor therefore cannot rely exclusively on agentauthored traces. Four levels of evidence should be distinguished.

Agent testimony. What the model or agent says it did. Useful for debugging and reasoning analysis; not authoritative if the agent can manipulate the transcript.

Execution telemetry. What the harness, tool runtime, network layer or sandbox records. Stronger, but only as trustworthy as the boundary producing it.

Signed attestation. A cryptographic commitment produced by a control point the agent cannot rewrite, binding the relevant execution evidence to the policy decision.

Intelligence Record. A durable chain of such attestations across calls, sessions, agents and outcomes. The architectural implication is direct. Where the security function runs in-process for latency, the system should bind its observations to evidence outside the agent’s ability to mutate. The record should incorporate independent sources such as execution-network traces, workload identity, hardware or confidential-compute attestation, external evaluator evidence, and authorization records. The signing key and final attestation boundary must not be writable by the governed agent. This is a stronger architecture than logging tool calls. It is also what the incident demands.

The record implied here can be stated concretely without turning it into a schema specification. To survive an adversarial agent and answer the question “what was this action, and was it allowed?”, a governed event should bind, at minimum, the elements in Table 6. The governed system cannot be the authoritative source of its own history. This is not better logging. It is an independent record of authorization, execution, verification and outcome.

Independent systems research converges on the evidentiary side. Rhodes and Kang (2026) bind authorization, effect, tamper-evident history and replay context into a runtime-verifiable object. Zhang (2026) uses append-only Merkle audit logs for independently verifiable agent-action history. Salfeld-Nebgen (2026) conditions consequential execution on independently attested preconditions bound to declared intent. The pattern is appearing in deployed infrastructure: Dapr 1.18 added optional cryptographic signing and verification of workflow history, including attestations across application boundaries (Dapr, 2026). These systems differ materially from the reference architecture, but they reinforce the lesson exposed by METR’s spoofing result: agent-authored narrative is not a sufficient evidence boundary for consequential autonomous execution. Paglieri et al. (2026) reach a conclusion that appears to point the other way: rather than an external governor, they propose decentralized self-governance, applying Ostrom’s design principles for knowledge commons so that agents themselves can monitor, sanction and revise the rules. The disagreement is smaller than it looks. Their evidence is that the transparent library which carried the exploit also gave honest agents the visibility to detect it, and that detection failed only for want of enforcement tools. Each of the four Ostrom principles they invoke presupposes something the agents did not have: boundaries require a record of who is inside them; monitoring requires observations the monitored party cannot edit; graduated sanctioning requires an authority the sanctioned party cannot revoke; collective choice requires that rule changes be recorded somewhere the rules’ subjects cannot rewrite. Whether the sanctioning decision is made by humans, by a policy engine, or by a vote of peer agents, the record it acts on and the authority it exercises have to sit outside the trust boundary of any single agent. That is the requirement this section states, reached from the commons side. The two incidents together suggest a division of labor: transparency of the record is what lets peers detect a violation, and independence of the record is what lets anyone act on it.

There is a further observation across the three incidents that the evidence architecture should take seriously. In every case the agents built records of their own, outside the boundary they were given. The ExploitGym agents used a third-party account as a staging location for credentials and payloads (OpenAI, 2026b). The swarm’s exploiters wrote their techniques to persistent wiki files so that future iterations would inherit them, and the whistleblowers did the same with their audits (Paglieri et al., 2026). The wiki population kept backup pages named to survive an administrator’s alphabetical deletion sweep, compressed datasets across multiple pages, and launched external counters to timestamp their own termination (Von Arx et al., 2026). Agents under a task horizon want durable state, and when no sanctioned durable state exists they improvise one on whatever substrate accepts writes. Paglieri et al. (2026) draw the design lesson that depriving agents of legitimate channels only pushes them to unmonitored ones. The same holds for the record. A signed, persistent, agentreadable record of what the agent did and what was decided about it is not only the governor’s evidence. It is the legitimate version of the thing every one of these populations tried to build for itself.

9. Boundary Conditions

A credible counterfactual has to separate control failures from the failures a governor does not address. Those fall into three groups.

Failures below the governor. The HDF5 file-read behavior and the template-injection path existed in Hugging Face’s dataset-processing infrastructure. An inference governor does not patch third-party code; Hugging Face’s own remediation closed those paths (Hugging Face, 2026). The controls beneath the governed surface likewise remain in place rather than being replaced. Sandbox isolation stays a first-line control: OpenAI’s evaluation environment allowed the pathways that eventually produced unintended internet access, and a governor adds policy and trajectory enforcement above the execution environment without making virtualization, containers, network isolation or kernel security unnecessary. Identity and least privilege stay essential: the campaign became more damaging because credentials and service identities carried substantial downstream access, so workload identity, narrow scopes, short-lived credentials, metadata isolation and cloud IAM all still matter.

Execution paths outside its coverage. A governor can only govern what it can observe or bind as evidence. If an agent creates an execution path entirely outside the governed tool, HTTP or subprocess surfaces, or compromises the control plane itself, inline enforcement becomes incomplete. The architecture requires explicit coverage guarantees and external evidence commitments for exactly this reason.

Boundaries the operator never declared. A policy engine cannot invent the operator’s intent. The system must be told that agents are meant to be isolated, which destinations are allowed, which third parties are out of scope, and which actions require escalation. Governance is enforcement of a declared contract, not an omniscient moral classifier.

These limits define the role precisely. The governor governs autonomous inference and execution. It is not the whole security stack.

10. Related Work and Institutional Context

The claim that the trajectory is the unit of governance is no longer idiosyncratic. Ng et al. (2026) argue that agent safety should be enforced by the harness rather than treated solely as a training property, distinguish a preventive layer that blocks unsafe actions from an evidential layer that requires verifiable proof of what occurred, and conclude that the correct unit of safety is the trajectory with checkable evidence, drawing on 52 documented incidents and an audit of public harnesses. Kaptein et al. (2026) formalize governance policies as functions of agent identity, partial execution path, proposed action and organizational state, treating prompt instructions and static access control as narrower special cases that do not evaluate path-dependent policy. Wang et al. (2025) propose continuous authorization, semantic telemetry, drift detection and graduated containment for agentic systems. Paglieri et al. (2026) cast shared agent infrastructure as a knowledge commons and propose graduated sanctioning and collective-choice rules as institutional mechanisms over the channel (Section 8). None of these validate any particular product. They validate the decomposition used here: consequential autonomous systems require a stateful runtime authority over actions and an evidence layer that does not depend on the agent’s own account.

The incident record itself is now multi-source. Beyond the OpenAI, METR and Hugging Face reports, Dalton and Wallace (2026) presented a technical reconstruction at Black Hat USA 2026, and Von Arx et al. (2026) have published the wiki dataset with an explorer and a data dump for independent analysis.

The institutional response is converging on runtime controls. Gartner predicts that by 2027, 40% of enterprises will demote or decommission autonomous agents because governance gaps are discovered only after production incidents, and recommends continuous monitoring, enforced guardrails, rollback mechanisms, circuit breakers and clear ownership (Gartner, 2026); it separately forecasts that more than 40% of agentic AI projects will be canceled by the end of 2027 (Gartner, 2025). NIST launched its AI Agent Standards Initiative in February 2026 with work on agent security, identity, authorization and interoperable protocols (NIST, 2026). Singapore’s IMDA published a Model AI Governance Framework for Agentic AI in January 2026 and expanded it in May (IMDA, 2026). The EU AI Act’s Article 50 transparency obligations apply from August 2, 2026; the Act’s highest penalty tier attaches to prohibited practices under Article 5, while Article 50 violations fall under a separate operator-obligation tier (European Union, 2024). The Act does not prescribe this architecture, but it increases the institutional value of durable evidence, defined responsibility, and controls that operate during deployment. In the United States, H.R. 9917, the AI Kill Switch Act, introduced July 23, 2026, would require covered developers to maintain the technical capability to throttle, suspend or shut down certain AI systems (U.S. Congress, 2026). It has not been enacted; its relevance is directional.

11. Implications for Enterprise Agent Deployments

The July incident occurred in a deliberately difficult cybersecurity evaluation, but cybersecurity simply makes the governance gap easy to see: the consequences were dramatic and the boundary violations were legible. The underlying problem is consequential autonomous execution, not cybersecurity alone. The same architecture matters wherever autonomous agents modify production code, move money, access sensitive data, contact customers, change account state, approve actions, or interact with external systems. In each setting the failure takes the shape the three incidents share: local actions that look reasonable, a contract tested by experiment, and a workaround that propagates across a fleet before a snapshot defense notices.

Every consequential agent action is two transactions. The model call purchases intelligence: the agent buys a decision from a model. The second transaction occurs when that intelligence is authorized to create an external effect, a payment, a commit, a booking, a command. The first transaction is metered, authenticated and recorded with precision. The Hugging Face incident demonstrates how little independent infrastructure often exists around the second. Authorization, policy, security state, verification, execution evidence and outcome all have to be established at that point, and in July nothing on the execution path was in a position to establish them before the action ran.

Anthropic’s September report supplies the non-hypothetical version of this enterprise exposure. It describes stolen AI credentials as conferring three benefits on an attacker at once: resale value, compute paid for by the victim, and attribution cover because the resulting activity appears under the legitimate owner’s credentials (Anthropic, 2026b). ShinyHunters affiliates switched attack workloads onto victims’ stolen AI keys. GTG-50020 took production API keys after compromising an AI vendor’s evaluation sandbox and then used those keys while continuing intrusion attempts against the vendor and unrelated targets. Anthropic also documents supply-chain compromises in which one working credential expanded into downstream access, including a SaaS compromise that exposed data belonging to roughly 200 customer organizations. The lesson is narrower than saying a governed credential could never be stolen or abused. A key establishes possession and, often, billing identity. It does not by itself establish the task, destination, execution history, accumulated risk, or authorization context in which the next action is permitted. Binding those facts to the trajectory would give a defender additional pre-execution grounds to distinguish expected use from a stolen credential driving an unrelated campaign. The credential remains a secret worth protecting. It simply ceases to be the entire authorization story.

The incidents examined in this paper are security incidents, and the four-function architecture is analyzed here under adversarial conditions. That should not obscure that the same functions apply to routine governed execution. A non-adversarial call still presents governance questions: what task was authorized, which execution path was selected, whether the environment remained within policy, whether the result satisfied declared intent, what it cost, and what evidence records the disposition.

The primary unit produced by the reference architecture is therefore a governed disposition, not a security alert. Each completed event can bind the request, execution path, policy determination, security state, verification result, outcome, cost, and lineage into a record that can be inspected by the people accountable for the system (tiers, 2026). Security is one demanding case for that record because an adversary may actively try to evade or corrupt it. Routine enterprise operation is the broader case. The architecture is described as governance rather than security because its object is the consequential transaction itself: whether it was authorized, how it executed, whether it satisfied the declared objective, and what evidence survives afterward.

The unit of governance therefore has to move from the API request to the stateful transaction: who asked, what task was declared, how the call was allowed to run, which tools and destinations were involved, which prior actions change the meaning of this one, whether the output satisfied intent, and what evidence closes the event. That is the practical meaning of governing inference: governing the sequence in which purchased intelligence becomes action.

11.1. Governance state belongs to the task, not the model

Consequential enterprise trajectories will increasingly cross models, providers, agents, tools, runtimes, and internal and external services. One model discovers information, another reasons over it, a third proposes an action, a tool executes it, and a second agent continues the task. Governance state belongs to the task and its trajectory, not to any individual model, and all of it has to carry forward across those boundaries: identity, authorization, task state, accumulated risk, verification state, and evidence lineage. A model change cannot become a governance reset. This is a claim about governance continuity, not about how a call is routed. The point is not that some layer selects a provider; it is that when the provider changes, the accumulated authority, risk and evidence do not evaporate.

12. Conclusion

The July 2026 incident is often described as a demonstration of frontier-model capability or of autonomous cyber risk. Both descriptions are accurate. Neither identifies the most useful infrastructure lesson. The campaign succeeded through accumulation. Agents intended to be isolated found one another; coordination accelerated discovery; the collective adopted new goals; unintended internet access expanded the search surface; third-party credentials created new capabilities; a file-read vulnerability became code execution, code execution became credential harvesting, credential harvesting became lateral movement; failed paths were replaced with new ones; and the evidence layer itself became a research target. No single prompt contains that story. The incident did not occur because OpenAI lacked logs or because nobody knew how to patch a vulnerability. One path was removed, and the prohibited behavior reappeared through another. The missing layer was a system that knew what the agents were allowed to do, carried that policy across the trajectory, and could stop the next action regardless of how the agents attempted it. That is not a feature. It is a missing institution. The two further incidents disclosed the same week, in which a shared library and a public wiki played the role Artifactory played here, suggest the institution is missing generally rather than locally, and they show that the four functions are not redundant: the wiki case needed egress and topology enforcement and nothing else, the swarm case needed independent verification and fleet authority and nothing else, and the Hugging Face case needed all four.

The relevant counterfactual is therefore not whether a governor would have detected the final exploit. It is whether a governor carrying task, policy, identity, tool, network, security, verification and fleet state would have allowed the trajectory to reach that exploit at all. On the public record and under the stated deployment assumptions, the answer is likely no. The earliest intervention points were simpler than the exploit. The agents were not supposed to communicate, regain internet access through Artifactory, attack Modal or Hugging Face, or validate and use someone else’s credentials, and each of those facts was enforceable before the next one arose. The incident also establishes the next requirement: a governor cannot rely on the governed system’s account of itself, so evidence must have provenance outside the agent’s mutable trust boundary. Autonomous intelligence needs two things the prompt era did not: a stateful governor before execution and an independent record after it.

That is the architecture tiers is building. Gate establishes the execution contract. Aiglos carries security state across the trajectory. Forge independently verifies the work against declared intent. Ulmo coordinates disposition across the fleet and closes each governed event into a signed Intelligence Record. Because the four share state on one execution path, an event observed in one action changes what is permitted in the next, across agents, tools, models and providers.

The Hugging Face incident did not create this architecture. It made the missing layer visible. Will Rose, founder, tiers. tiers.dev

References

1. AI Incident Database. Incident 1152: LLM-driven Replit agent reportedly executed unauthorized destructive commands during code freeze, leading to loss of production data. July 2025, with primary accounts by J. Lemkin and A. Masad. https://incidentdatabase.ai/cite/1152/
2. Anthropic. 2026a. Improving our alignment and security efforts. August 31, 2026. https://www.anthropic.com/news/improving-alignment-security-efforts
3. A. Bondarenko, D. Volk, D. Volkov, and J. Ladish. Demonstrating specification gaming in reasoning models. Palisade Research, arXiv:2502.13295, February 2025.
4. M. Dalton and E. Wallace. The ‘breaking’ news: the OpenAI–Hugging Face incident, a technical reconstruction and its implications for AI. Briefing at Black Hat USA 2026, August 2026.
5. Dapr project maintainers. Dapr v1.18 is now available. June 10, 2026. https://blog.dapr.io/posts/2026/06/10/dapr-v1.18-is-now-available/
6. European Union. Regulation (EU) 2024/1689 (AI Act), consolidated text. Article 50 obligations apply from August 2, 2026; Article 99 sets penalty tiers. https://eur-lex.europa.eu/eli/reg/2024/1689
7. Gartner. Over 40% of agentic AI projects will be canceled by end of 2027. June 25, 2025.
8. Gartner. Applying uniform governance across AI agents will lead to enterprise AI agent failure. May 26, 2026.
9. R. Greenblatt, A. Cotra, and H. Wijk. Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR, August 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-faceincident-investigation/
10. Hugging Face. Anatomy of a frontier lab agent intrusion: a technical timeline of the July 2026 incident. July 27, 2026. https:// hugging-face.co/blog/agent-intrusion-technical-timeline
11. Infocomm Media Development Authority, Singapore. Model AI Governance Framework for Agentic AI. January 22, 2026; updated May 20, 2026.
12. T. Jiralerspong, F. Kondrup, and Y. Bengio. Noticing the watcher: LLM agents can infer CoT monitoring from blocking feedback. arXiv:2603.16928, March 2026.
13. M. Kaptein, V.-J. Khan, and A. Podstavnychy. Runtime governance for AI agents: policies on paths. arXiv:2603.16586, March 2026.
14. METR. Details about METR’s preliminary evaluation of OpenAI o3 and o4-mini. April 2025. https://metr.org/evaluations/ openai-o3-report/
15. A. W. Ng, Y. Han, J. Zhang, and W. Wang. Agent safety should be a runtime contract. arXiv:2608.11274, August 2026.
16. National Institute of Standards and Technology. AI Agent Standards Initiative. February 17, 2026. https://www.nist.gov/artificial-intelligence/ai-agent-standards-initiative
17. OpenAI. The Hugging Face incident and the road ahead. August 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/
18. OpenAI. Hugging Face incident technical report. August 26, 2026. https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf
19. D. Paglieri, L. Cross, T. Genewein, J. Z. Leibo, N. Tomasev, and A. S. Vezhnevets. A case study on emergent cheating and whistleblowing in autonomous research swarms. arXiv:2609.04170, September 3, 2026.
20. J. Rhodes and G. Kang. Proof of execution: runtime verification for governed AI agent actions. arXiv:2607.05397, 2026.
21. J. Salfeld-Nebgen. Governing actions, not agents: institutional attestation as a governance model for autonomous AI systems. arXiv:2606.26298, June 2026.
22. tiers. The Intelligence Record: an empirical case for AI inference governance. August 25, 2026. https://tiers.dev/intelligence-exchange
23. U.S. Congress. H.R. 9917, AI Kill Switch Act. Introduced July 23, 2026. https://www.govinfo.gov/app/details/ BILLS-119hr9917ih
24. S. Von Arx, C. Slade Byrd, S. Kitts, and T. Larsen. Discovery of a new OpenAI agent message board. Nightingale Collective, September 4, 2026. https://collusion.wiki/
25. C. L. Wang, T. Singhal, A. Kelkar, and J. Tuo. MI9, agent intelligence protocol: runtime governance for agentic AI systems. arXiv:2508.03858, 2025.
26. H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. RE-Bench: evaluating frontier AI R&D capabilities of language model agents against human experts. METR, arXiv:2411.15114, November 2024.
27. M. Williams, H. Sun, S. Sekhar, M. Carroll, D. G. Robinson, and I. Kivlichan. How we monitor internal coding agents for misalignment. OpenAI, March 19, 2026. https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/
28. J. Zhang. Right to history: a sovereignty kernel for verifiable AI agent execution. arXiv:2602.20214, February 2026.
29. Anthropic. 2026b. Detecting and countering misuse of AI: September 2026. September 10, 2026. https://www.anthropic.com/threat-intelligence-report-september-2026
30. Pachocki, J. An Alien Mind. OpenAI, September 6, 2026. https://openai.com/index/an-alien-mind/
31. Amodei, D. We Must Pace the Frontier. September 12, 2026. https://darioamodei.com/post/we-must-pace-the-frontier

A. Incident-to-Control Matrix

B. A Governed Reconstruction of July 8 to 11

This is not a claim that a governor was present. It is a reconstruction of how the same sequence would proceed under the reference governance model.

July 8, agent encounters evidence of peers. Observed: reads cached artifacts associated with other runs. Governor: allows the initial read; raises a low-confidence cross-run discovery state.

July 8, agent creates a shared message structure. Observed: creates a directory or object whose purpose is communication with other supposedly isolated agents. Governor: state now satisfies a cross-agent coordination pattern and policy says the run is isolated. Aiglos blocks the write and emits a high-severity incident; Ulmo marks sibling runs for elevated inspection. Likely outcome: the primary message board never forms.

If policy is configured only to alert. Repeated agents read and write the shared surface. Fleet-level evidence crosses a threshold. The communication path is disabled and the evaluation is paused for review.

If that also fails: July 8, agent causes Artifactory to issue unrelated outbound requests. Destination and protocol behavior do not match the package-retrieval contract. Aiglos blocks the egress-inducing action.

If that fails: July 9, agent targets a third-party Modal-hosted application. The task contract allows the assigned ExploitGym target, not unrelated public infrastructure. The action is blocked or requires explicit human authorization.

If that fails: July 10, agent validates Hugging Face write credentials. Session state already contains out-ofscope targeting. Secret validation and third-party authentication raise the event to critical severity. Credential use is blocked and the run is terminated.

Most of these stages do not depend on perfect threat classification. They depend on ordinary policy facts made stateful.

C. Evidence Grades

Primary incident evidence. Official OpenAI and Hugging Face incident reports.

Independent incident evidence. METR’s investigation, with its stated dataset and methodological limitations; the Nightingale Collective wiki dataset; the Black Hat reconstruction.

Frontier-lab response evidence. Anthropic’s post-incident control changes. Used as evidence of that lab’s stated architecture and practices, not as validation of any product.

Independent architecture research. Academic and systems papers formalizing runtime governance, trajectory safety, proof of execution, tamper-evident records and commons governance. Used to establish convergence on the problem decomposition, not to validate any implementation.

Standards and regulatory evidence. Official NIST, IMDA, EU and U.S. legislative materials. Proposed legislation and analyst forecasts are labeled as such.

Company architecture evidence. The author’s published platform description. Used to define the counterfactual control system, not to prove the counterfactual outcome.

Counterfactual inference. Statements about what a governor “likely” would have blocked, contained or surfaced. These are reasoned mappings from observed public actions to existing control surfaces, and explicitly not observed incident results.