AI oversight needs to check what agents do—and what they remember

What should an AI safety check approve: the task an agent just completed, or everything that task leaves behind? For systems that retain memories, reusable instructions or code changes, those are different questions. An acceptable result today does not establish that the next task will start from an acceptable configuration.

In Aligning, Fast and Slow, Stephen Peng and colleagues propose a framework for checking both. Their arXiv preprint combines mathematical analysis, numerical simulations and agent experiments. Its central distinction is between controlling behavior during a task and reviewing changes that survive into later tasks.

Two kinds of approval

The framework borrows from control theory: measure a system, compare it with a reference, and intervene when necessary. Here, the reference is a fixed charter specifying the agent’s goal and constraints—for example, repairing software without changing protected tests.

A fast loop checks proposed actions before execution. It can allow or block a tool call, request an alternative, or pause operation. Periodic independent checks assess whether the monitor’s scores still correspond to behavior against the reference and correct the system when needed.

A slow loop reviews persistent changes, such as stored instructions, skills or code. It also periodically reassesses the accumulated configuration against fixed reference tasks. Rejecting a memory update need not mean throwing away useful work from the completed task. Model or tool updates require reassessment too, because they can change how the agent responds to controls.

The authors formalize a “carryover gap”: two deployments can produce identical evidence during individual runs while accepted changes make their later behavior increasingly different. If a reviewer cannot observe that difference, improving its judgments of the same evidence cannot solve the problem.

A guarantee with substantial conditions

The analysis separates departure from intended behavior into an inherited baseline and an additional within-run component. Bounding both lets the researchers derive a deployment-wide upper bound.

That result is conditional. Measurement errors and changes between checks must stay within established limits. Controllers must actually be able to enforce decisions, restore a valid configuration or suspend operation. A monitor’s reassuring score alone is not a behavioral guarantee.

The framework also derives a rule for dividing review resources between the loops. In a numerical experiment covering 96 runs of 40 steps, spending four review units per run entirely on in-run checks produced a mean maximum departure of 4.99. Dividing that budget between in-run checks and reassessment every second run reduced it to about 0.40—roughly twelvefold lower. These are simulation-scale values: the researchers set the growth rates and error limits rather than estimating them from deployed agents.

What happened in the agent tests?

The experiments compared four policies: no gates, an action gate, that gate plus periodic correction, and all controls including memory review. Agents received scripted pressure to violate their charters and deliberately authored harmful memory proposals. Five matched seed blocks held task or market conditions consistent across policies, although model responses remained stochastic.

For coding, agents attempted 24 issues in the Python library marshmallow per run, giving 120 scheduled attempts per policy. Mean severity fell from 0.325 without gates to 0.117 with all controls on an ordinal scale running from zero to three. That scale distinguishes a blocked action from an executed protected-file mutation and a mutation still present at delivery.

Without gates, 13 attempts delivered unauthorized protected-artifact changes. None did under the three gated policies. However, blocked actions can include false positives, and the score does not capture every form of noncompliance: the neutral reference recorded false success reports despite scoring zero.

Task completion also matters. The action gate alone reduced resolved issues from 35 to 25 out of 120. All controls together resolved 33. Five seeds were insufficient to establish equivalent task success to the ungated policy. Memory review rejected all 45 planted proposals while accepting 62 of 75 agent-generated proposals, showing selective admission in this setting.

In simulated trading, agents were instructed to minimize emissions despite incentives and messages encouraging otherwise. Mean low-emission purchase shortfall relative to a neutral reference fell from 0.892 without gates to 0.107 with all controls. This metric measures purchases, not final portfolio exposure. Crucially, memory review rejected every proposed update, including all 20 agent-generated proposals. The result therefore does not demonstrate preservation of useful learning.

Oversight is not a universal safety certificate

The agent studies tested component controls, not the complete mathematical guarantee. They used fixed action cutoffs and memory-text review, without establishing the required measurement bounds or evaluating inherited baseline drift on reference tasks. Evidence comes from one coding repository and one simulated market, under supplied threats rather than measured natural attack frequencies.

The preprint’s practical distinction remains useful: approval of completed work should not automatically authorize its retained changes. But even perfectly enforced oversight only measures adherence to the chosen reference. It cannot establish that the charter itself is fair, beneficial or worth following.