A Robot Memory System That Keeps the Lesson, Not Every Detail

From recordings to reusable knowledge

Remembering more is not necessarily remembering better. A stored robot interaction includes useful information about an object and an action, but also incidental details about the surrounding scene. HiMEM’s developers argue that this clutter can obscure patterns that transfer to another task.

Boran Zhao and colleagues at Xi’an Jiaotong University describe their approach in an arXiv preprint posted on November 6, 2026. HiMEM extends their earlier framework, ECHO. The supplied source identifies this version as a preprint; it does not establish peer-reviewed publication of HiMEM.

The system supports vision-language-action models: software that combines camera observations and language instructions to generate robot actions. Its central idea is to maintain both specific experiences and compact, conditional lessons drawn from them.

The brain connection is an inspiration, not evidence of biological equivalence. The authors borrow from complementary learning systems theory, which distinguishes rapid storage of particular experiences from the gradual development of more general knowledge.

How the memory works

HiMEM first divides recorded executions into smaller subgoals. A vision-language model, Qwen2-VL-7B-Instruct, proposes these divisions. Whether an action actually succeeded is checked separately, using simulator states or human-verified physical execution records. This distinction matters: failing an entire task does not mean every preceding action failed.

Individual experiences are organized in a branching hierarchy, from general tasks to more specific subgoals and actions. The framework uses hyperbolic geometry—a mathematical space suited to representing hierarchical relationships—to search this memory.

Offline, a vision-language model compares experiences and proposes rules linking conditions, operations and estimated success. Verified successes and failures supply the underlying counts. Each rule retains references to its source evidence; unresolved or conflicting outcomes are withheld from the statistics.

During execution, the system retrieves successful examples and uses applicable rules to strengthen, weaken or block their influence. Failed experiences contribute evidence about risk, rather than action examples to imitate. A small correction derived from memory then enters the underlying model’s action-generation pathway.

HiMEM can also combine compatible successful memory representations. These temporary combinations guide the current execution but are not stored as verified experiences.

Forgetting without erasing the evidence

Once a rule captures a recurring pattern, some individual memory vectors become redundant. HiMEM removes eligible duplicates while protecting distinctive failures, cases near condition boundaries and other unusual evidence. It retains a compact record of provenance and outcomes, so pruning does not rewrite the rule’s success statistics.

That safeguard does not make forgetting cost-free. In one controlled LIBERO-Long experiment, removing 25.2% of episodic vectors reduced success from 95.3% to 94.9%. Removing 56.6% reduced it to 92.9%. The results show a storage–performance tradeoff, not that progressively smaller memory is always better.

What the experiments show

Across four suites of the LIBERO simulated manipulation benchmark, the authors report that adding HiMEM raised the pi-zero model’s average success from 91.88% to 98.20%: a gain of 6.32 percentage points. Compared with ECHO’s 97.30%, the additional gain was 0.90 points. Two other underlying models also improved.

The matched comparisons are particularly informative because the authors report holding policy checkpoints and evaluation conditions constant. They also separated memory-building executions from testing and froze memory during evaluation.

To examine transfer, the team excluded target-suite executions from both episodic memory and rule construction. In that LIBERO cross-suite setting, HiMEM reached 92.78%, compared with 89.31% for ECHO and 80.70% for the underlying model. This tests reuse without direct target experience in memory, rather than unrestricted generalization.

Physical experiments used a Franka Emika Panda arm. Across five held-out composite tasks, each with 20 matched starting configurations, average success was 61% with HiMEM, 47% with ECHO and 37% without either memory framework. The tasks recombined operations such as placement, temporary storage and object transfers.

Where the evidence stops

These are promising results within defined benchmarks and a limited physical setup, not proof of dependable household robotics. A 61% composite-task success rate still leaves substantial failure.

The method depends on recorded experience, reliable outcome verification and useful condition descriptions. Sparse evidence limits the support for its rules, while video analysis and consolidation add offline computation. Reported memory-search latency should not be mistaken for total system latency.

The supplied HTML extraction also omits figures and may distort tables and equations. Several reported scores differ slightly across experimental tables, making named comparisons safer than treating every result as one interchangeable measurement.

The contribution is therefore specific: HiMEM offers evidence that organizing experience into examples and conditional lessons can improve robot manipulation. It does not show that a robot has acquired human-like memory—or that its lessons will remain reliable outside the conditions tested.