AI agents can encode better options and still choose worse ones

An agent navigating a maze might discover an obstacle, retreat, then repeat the same unsuccessful route. That behavior does not, by itself, reveal what went wrong. The model might lack a useful map, misjudge its options or fail to translate a useful evaluation into action.

A preprint by Yuzhang Luo, Mohamed Aghzal, Liangming Pan and Ziyu Yao investigates the last possibility. Across three language models and two simulated environments, the researchers found internal signals that distinguished better actions from worse ones—even when the agent preferred the worse action. They call this value–policy dissociation: a gap between represented action quality and the policy, meaning the model’s rule for choosing actions.

Looking inside a decision

The team studied Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct and GPT-OSS-20B. Their first environment contained 1,400 five-by-five grid maps. Agents could observe adjacent cells and received a maintained record of previously discovered terrain. They had 24 moves to reach a goal.

The second environment, WebShop, supplied 1,502 simulated shopping tasks. Agents searched and clicked through products, receiving a final reward between zero and one according to how well a purchase matched the request.

To investigate internal evaluations, the researchers trained linear probes: simple mathematical readouts that extract a target signal from a model’s hidden activations. These readouts were fitted using training examples and evaluated on separate data.

Defining a good action required different approaches in the two environments. In Grid, the target was the shortest remaining route after a move, calculated from the currently observed map. Unknown cells were treated as traversable. This was therefore an optimistic distance estimate, not privileged knowledge of hidden obstacles.

In WebShop, the researchers estimated a product click’s value by averaging the final rewards from 40 sampled continuations after that click. The probe learned to rank pairs of candidates. Search queries were outside this analysis.

A readable signal does not guarantee a good choice

The probes recovered useful action rankings from intermediate model layers. Grid readouts could account for obstacle-induced detours rather than merely favoring moves geometrically closer to the goal. WebShop readouts achieved around 80% pairwise ranking accuracy.

Crucially, correct readouts frequently accompanied incorrect policy preferences. Across the three models, the probe favored the better action while the policy favored the worse one in 34.2%–58.3% of evaluated Grid states. The corresponding WebShop fractions were 39.8%–43.9%.

These percentages describe evaluated decisions, not the proportion of complete tasks that failed. They also do not establish human-like understanding. They show that information predictive of the researchers’ action-quality targets was recoverable before the model generated its next action.

Turning a readout into an intervention

Could that information help change behavior? Simply pushing internal activations along the probe’s readout direction produced limited and inconsistent effects. Making a signal more readable was not reliably equivalent to making it guide an action.

The researchers instead developed Natural Value Transport, or NVT. It learns how activations change alongside differences in action value. In Grid, training comparisons changed a cell between blocked and free while holding other task conditions fixed. In WebShop, they compared candidate-product representations associated with different continuation rewards.

NVT then used those learned relationships to construct activation edits. Compared with magnitude-matched probe, shuffled and random controls, it more consistently shifted action preferences in the intended direction, especially in later layers.

A controller combined the value probe as a sensor with NVT as an actuator, reconsidering interventions during interaction. On 100 held-out Grid maps, Qwen2.5-7B’s success rose from 14% to 30%; the other two models also improved.

On 454 eligible WebShop episodes for Qwen2.5-32B, mean reward increased from 0.358 to 0.452. These were selected trajectories containing qualifying product-click decisions, not the full original benchmark. The other models also gained reward, while reductions in shopping loops were modest.

What remains unresolved

The study is a preprint, and its conclusions remain provisional. Three models from two families and two environments cannot establish a universal explanation of agent failure.

The value targets also have boundaries: Grid distances assume unknown terrain is open, while shopping estimates depend on finite samples and the model’s subsequent behavior. Linear readouts and interventions do not reveal the complete mechanism by which evaluations arise or influence outputs.

The controller requires internal activation access and task-specific training data. Grid evaluation also replaces proposed moves into known obstacles or outside the map with a probe-selected valid move. Steering did not consistently reduce invalid outputs.

The central result is narrower than a general planning fix: an agent’s poor choice need not mean that all useful information about better alternatives is absent. Reading that information and making the model use it are distinct technical problems.