A Preprint Proposes Turning AI Agents’ Past Attempts Into Training Rubrics

The problem with a single outcome

When a language-model agent shops on a simulated website or carries out a household task, it produces a sequence of actions and receives observations along the way. Training can reward the final result, but that result does not reveal which parts of the sequence helped and which were mistakes.

One common approach, Group Relative Policy Optimization (GRPO), compares outcomes across attempts at the same task. In its trajectory-level form, however, the resulting learning signal is shared by the model-generated tokens throughout each attempt. A useful product check and a redundant action can therefore receive the same broad credit.

The preprint proposes Online Rubric Self-Distillation, or ORSD, as an addition to that outcome-based signal. Its central idea is to reuse a group of attempts twice: first to measure their outcomes, and then to derive written criteria that help shape training at the token level.

From attempts to criteria

At each training update, the current agent generates several attempts at the same task. It then receives the task description and those attempts and writes multiple candidate bundles of short process criteria. For a shopping task, a criterion might call for checking a product’s price before purchasing it or avoiding a choice based on only a partial match.

The researchers do not give the outcome rewards to the agent when it generates the rubrics. They use those rewards afterward to choose among the candidates. A separate, frozen verifier scores each attempt against each criterion. The scores within a bundle are averaged for each attempt, and the researchers select the bundle whose scores align most strongly with the ranking of the attempts by reward.

This selection favors criteria that distinguish higher-reward attempts from lower-reward ones. It does not prove that every criterion captures a causal reason for success. In WebShop, the training reward includes the official task score and success, with penalties for invalid and repeated no-progress actions; in ALFWorld, it is a binary success indicator.

Training with privileged context

The selected rubric is privileged context: extra information used during training rather than a checklist the agent must consult while performing the task. The researchers replay the already sampled model outputs twice using the same policy—once with the rubric in the context and once without it.

For each model-generated token, ORSD measures the difference between its log-likelihood in the two contexts. If the rubric makes a token more likely, the method increases that token’s learning advantage; if it makes the token less likely, the advantage decreases. A scaled version of this difference is added to the original GRPO advantage. Environment observations are excluded from the token correction.

The comparison requires no new environment rollouts, although rubric generation, verifier scoring and replay require additional computation. Nor is the likelihood difference a direct judgment that a token is correct: it measures how the selected rubric changes the policy’s probability for that token.

What the experiments found

The authors evaluated ORSD on ALFWorld, an interactive household-task benchmark, and WebShop, a simulated shopping benchmark, using Qwen3-1.7B and Qwen3-4B models. Their GRPO comparison used the same training data, reward definitions and environment-interaction budget.

On ALFWorld, reported overall success for the smaller model rose from 53.92% with GRPO to 66.04% with ORSD; for the larger model, it rose from 74.07% to 77.43%. On WebShop, success rose from 55.40% to 63.00% for the smaller model and from 66.15% to 77.05% for the larger one. ORSD also had the highest overall results among the methods compared in the paper, though some published baselines retained their own evaluation protocols.

WebShop ablations lend support to parts of the design. Rubrics generated once before training improved on GRPO but fell short of the online version. Selecting among candidate bundles by reward alignment also outperformed random selection in the reported tests. These comparisons support the approach in the tested settings; they do not establish that each component will help equally on other tasks.

The evidence boundary

The authors identify two unresolved issues. ORSD generates fresh rubrics for each update without retaining a repository of useful criteria. And selecting a rubric because it correlates with whole-trajectory rewards does not directly establish that it assigns credit well to individual decisions.

The preprint’s contribution is therefore a training mechanism and a set of benchmark results, not a demonstration that agents have learned a general theory of their own mistakes. Its practical proposition is narrower: existing attempts can supply process guidance that improves outcome-based training in the environments tested.