ENGRAM tests when AI agents can stop retrieving past experience

Two decisions instead of one

Language-model agents can use experience in two ways: retrieve stored examples while solving a task, or update their model parameters through training. ENGRAM separates the decision to train on an experience from the decision to stop retrieving it.

Its Engram Trace Graph organizes records by skill, outcome and source, alongside relationships between prerequisite skills. Records contain action sequences or solutions, their task conditions and evidence of success or failure.

A skill can enter training when the student demonstrates sufficient mastery and meets other eligibility conditions, or when enough successful teacher demonstrations are available. This second route matters because a student may fail at an earlier step and never reach the skill it needs to learn. Teacher demonstrations can therefore provide training material before independent mastery. The configured demonstration threshold counts successful stored records, not necessarily independent teacher episodes.

Checking whether the weights can replace retrieval

ENGRAM trains a LoRA adapter—a relatively small set of adjustable model parameters—using selected examples with graph-supplied context. It reserves examples for verification rather than including them in the current training batch.

After training, a gate compares the updated model without episodic retrieval against the original model with retrieval. Both receive the same task interface. Disabling episodic retrieval does not remove current observations, interaction history or benchmark-provided evidence.

Release requires actual participation in training, nonempty verification probes, a score meeting a relative comparison threshold and a score exceeding an absolute floor. Skills that fail retain external guidance. A separate check can approve an adapter for use alongside memory without authorizing release.

For interactive tasks, probes assess agreement with recorded actions. For question answering, they assess answer correctness against a threshold. These checks provide limited behavioral evidence, not proof of complete task competence.

What the experiments found

The authors evaluated Qwen3-8B and Llama-3.1-8B-Instruct across six benchmarks covering game achievements, household tasks, shopping, factual questions and financial reasoning.

In matched Qwen component comparisons, full ENGRAM exceeded the frozen-weight graph-memory control by 3.1–8.4 percentage points across the six benchmarks. ALFWorld success rose from 82.1% to 90.3%; HotpotQA answer F1 rose from 64.2% to 71.5%. F1 measures overlap between predicted and reference answer tokens.

Restricting training admission to student mastery reduced performance on every benchmark. Removing the adapter-activation gate also lowered scores, while eliminating retrieval after training produced larger declines. These results support retaining external guidance for some skills rather than assuming that all useful experience has transferred into parameters.

ENGRAM did not lead every comparison: its Qwen results were below the highest listed means on WebShop and FinQA. The baseline scores came from the authors’ implementations, including adaptations of some methods; sharing a backbone did not mean equal computational cost.

A controlled memory-release comparison

The clearest release-policy test used one fixed HotpotQA checkpoint and 600 experience records. Development probes approved release for three of five skills, leaving 269 records retrievable and archiving 331—a 55.2% reduction in the retrieval pool.

On held-out questions, retaining everything produced 71.1% F1, compared with 70.9% under verified release. The paired difference was −0.2 percentage points, with a 95% confidence interval from −0.6 to 0.2. That interval allows a small change in either direction; it does not establish zero performance loss.

Immediate release after training left only 13 retrievable records but reduced F1 to 66.8%. Random release at the same 269-record retrieval budget scored 67.6%. Within this experiment, choosing records through verification performed better than removing them indiscriminately.

All released records remained archived. The result demonstrates reduced retrieval availability, not reduced total storage or erased information.

Costs and limits

Efficiency depended on the task. Against the frozen-weight retrieval control, ENGRAM used 42.8% fewer input tokens and 13.6% fewer model calls on ALFWorld, but elapsed time increased from 43.76 to 65.91 seconds per task. On WebQuestions, it was 24.6% faster. Lower token usage did not consistently mean lower latency.

Verification also has coverage limits. Recorded-action agreement can miss valid alternative actions and downstream failures. The controlled release study used one checkpoint, so its uncertainty estimates do not capture variation across independent training runs. The scaled experiments used a single batch update, leaving longer consolidation sequences and changing task distributions as open questions.

Finally, behavioral verification does not establish privacy, safety or factual reliability. Removing retrieval access neither deletes archives nor removes information learned by the model. ENGRAM’s evidence supports selective withdrawal of external guidance under tested conditions—not guaranteed forgetting or universally safe memory removal.