Can an AI Reuse Advice from Similar Situations?

A robot learning to move or balance often needs many rounds of trial and error. Advice from a trainer can point it toward a useful action, but that help is less valuable if the system forgets it immediately. Persistent advising addresses this by saving advice; broad-persistent advising (BPA) also tries to reuse it in situations that resemble the original one.

In existing BPA, states are grouped using K-means clustering, and advice can be linked to a cluster. The authors argue that this can be a blunt tool, particularly when observations are high-dimensional: states placed in the same cluster might call for different actions. Their proposed alternative, Encoder BPA, learns a compact representation of recent observations and searches stored advice in that representation space.

How the method works

Before advice retrieval, the researchers train an encoder on sequences of states collected by a teacher agent. The encoder is trained to predict the teacher’s action, encouraging its representation to retain information relevant to choosing actions. It then maps a recent sequence into a latent representation—a numerical summary that can be compared with summaries already in memory. The paper uses a history of 16 states as encoder input.

When advice is available, the learner follows it and stores the encoded sequence together with the advised action. When new advice is unavailable, the system compares the current representation with stored ones. It can combine the actions associated with the five closest eligible memories: by majority vote for discrete actions, or by averaging for continuous actions. A consistency check can reject a candidate if the stored actions disagree too much. The system also retains a probabilistic advice-reuse mechanism.

The experiments compared methods in four simulated settings: Cart-Pole, Inverted Pendulum, MuJoCo Ant, and a partially observable version of Ant. The authors evaluated average reward over the most recent 100 episodes across nine random seeds, as well as how often stored advice was reused. The reported advisor configurations gave advice half the time, with either perfect accuracy or 90% accuracy.

What the study reports

Encoder BPA, using LSTM or Mamba encoders, produced the strongest early training performance in Cart-Pole. The authors also report early improvement in Inverted Pendulum and Ant, including the partially observable Ant task. The partial-observation result is consistent with the idea that a history of observations can offer useful context when a single observation does not reveal the full state. It does not, by itself, establish that this explanation caused the improvement.

The method generally reused advice more often than exact-state persistent advising, and it exceeded the original BPA in several—but not all—settings. For instance, the reported reuse counts were lower for the LSTM and Mamba versions than for BPA in the Ant setting. More reuse is not automatically better: the key question is whether the retrieved advice fits the current situation.

That distinction matters especially in continuous-action tasks. The authors report that averaging retrieved advice could introduce inappropriate actions and reduce rewards later in training. They also found sensitivity to similarity and consistency thresholds: changing them can substantially alter how often advice is reused. With less accurate advice, Encoder BPA remained sensitive to noise; the encoder helps find and reuse advice, but does not verify that the advice is correct.

What remains uncertain

These results come from four simulated control environments, not deployed robots or a broad range of real-world tasks. The experiments use particular encoders, advisor settings, and retrieval thresholds, so they do not show that one configuration will work generally. The authors also note that encoder performance can vary and that computing a representation at every step adds overhead. They suggest further work on more stable representations and on reducing reliance on stored advice as training progresses.

The paper’s conclusion is therefore narrower than “memory makes agents better”: latent retrieval can help reuse advice and improve early learning in these experiments, but poorly matched advice can become a liability. The source identifies the work as appearing in the Proceedings of the Australasian Conference on Robotics and Automation (ACRA), 2026. The version supplied here does not establish independent replication or broader validation; readers should treat the reported results as evidence from the study’s specific setup, not as a settled general rule.