AI agents remember useful lessons—but struggle to keep them current

Testing experience, not just recall

In a preprint posted to arXiv on November 3, 2026, Shijue Huang and colleagues introduce ReScopeBench, a benchmark containing 352 tasks across 18 simulated interactive environments. Scenarios include ticket booking, clinical-record corrections, filesystem operations and robot logistics. The reported findings are experimental results within this setup, not established evidence of reliability in deployed applications.

The benchmark separates two kinds of experience. Structural experience concerns where resources are and how they connect or can be accessed. Rule experience concerns reusable constraints governing valid actions. A delivery destination might move while its handling policy remains unchanged; an effective agent should revise one without discarding the other.

According to the authors, existing benchmarks examine memory, experience reuse and adaptation, but offer less visibility into which previously learned details agents correct, preserve or lose. ReScopeBench therefore evaluates saved knowledge alongside completed tasks, rather than treating task success as sufficient evidence of good memory maintenance.

Changes arrive without warning

The researchers built task sequences around shared, discoverable information, then introduced 26 controlled environmental changes. These included revised policies, moved resources, changed permissions and altered tool workflows. Agents received no explicit announcement: they had to detect changes through interaction.

Environment-specific resource budgets restricted repeated exploration. This matters because an agent that continually looks everything up might complete a task without maintaining useful experience, while spending resources needed for execution.

The team evaluated 20 models using the same Hermes agent harness. Each model ran two sequences per environment: one retaining saved memory, skills and retrievable task history, and another clearing those assets between tasks. Both started each task with a fresh conversation. Environment state persisted within each sequence and could diverge as agents took different actions.

Deterministic graders assessed delivered results. A separate language-model judge assessed saved experience against private reference claims and scoring rubrics.

Useful memory, incomplete updates

Every model in the results table achieved a higher task score with retained experience than without it. For example, Gemini-4-Argon scored 81.12 with experience versus 25.86 without, on a 0–100 scale. These are benchmark task scores, not a general measure of intelligence or real-world success rates.

Keeping information current proved harder. The highest immediate update-coverage score was 37.63: even the leading model correctly recorded fewer than two-fifths of the designated update targets after completing the first task affected by a change. That measure allows adaptation during the task; it is not a test taken before the agent can investigate.

Structural coverage exceeded rule coverage for 17 of the 20 models. The imbalance was not universal: Claude Opus 5.5 recorded rules more completely than structural information.

A separate analysis followed previously mastered information across selected changes. By the fourth affected task, replacement-content coverage reached 64.63%, while coverage of still-valid content was 91.07%. In this subset, correcting obsolete knowledge lagged behind preserving valid knowledge. However, the authors caution that the two kinds of content differed in complexity and when tasks exposed the relevant evidence.

Knowing does not ensure doing

In a request-level analysis limited to two environments, complete saved prerequisites were associated with higher delivery rates. Nevertheless, some requests failed despite complete prior knowledge, while others succeeded with incomplete prior records. The comparison was descriptive, not proof that saved knowledge alone caused the difference.

One clinical-record example illustrates the execution problem. Four corrections cost eight resource units within a 17-unit budget; each context lookup cost five. GPT-5.4 performed three lookups and delivered only one correction, despite having the required bindings saved. Other models completed all four with fewer checks. The example shows a trade-off between verification and execution, not a general argument against checking uncertain information.

What the evidence leaves open

Each model–environment pair had only one sequence per condition, so the study does not establish run-to-run variability. The environments were coded simulations with deliberately constructed dependencies and changes; broader deployment performance remains untested here.

Experience scoring also depended on language-model judgments. A second judge showed strong agreement, but the authors explicitly distinguish consistency under shared rubrics from independent validation of accuracy. Coverage measures tracked reference knowledge, not the correctness of everything an agent saved.

The central finding is therefore narrower than a universal model ranking: under these controlled conditions, retaining experience helped, but reliable long-lived assistance required both selective revision and effective execution.