Teaching robot models which visual details to keep

Which parts of a camera image does a robot need right now—and which might matter for what happens next? A new arXiv preprint introduces APS-VLA, a method for learning that selection from robot demonstrations. Its experiments suggest that processing fewer visual tokens can shorten model inference while keeping task success near the original policy’s level. That is an efficiency result, not evidence that the robot completes entire jobs proportionally faster or more safely.

Selecting information without retraining the whole policy

Vision-language-action models combine camera observations, language instructions and information about the robot’s own state to generate movements. Images become collections of numerical representations called visual tokens, each carrying information about part of the view. Processing hundreds of these tokens repeatedly adds computational cost.

APS-VLA inserts a small learned selector before the policy’s main action-generating computation. The selector scores tokens using their visual features, the instruction and proprioception—the robot’s measurements of its own state. It keeps a fixed number from each camera, preserves their original ordering and identities, and packs them into a shorter sequence. Language and robot-state inputs remain intact.

The pretrained policy’s parameters stay frozen. However, the system is not training-free: the selector must learn from demonstration trajectories, and different policy architectures need their own token-packing integration. Visual encoding also still occurs; the method reduces the visual sequence processed downstream rather than eliminating the work of interpreting camera images altogether.

Learning from actions and their consequences

The selector receives two kinds of training feedback. First, the frozen policy operating on selected information should reproduce both its own predictions with the full visual input and the actions recorded in demonstrations. This encourages pruning that preserves useful control information rather than merely retaining visually prominent patches.

Second, a temporary prediction module uses the selected current representation and demonstrated actions to predict a compact feature summary of a later observation. The default horizon is eight demonstration steps. The target is not a reconstructed photograph: it is a compressed spatial summary of visual features.

This second objective asks whether the retained information helps explain how the observed scene changes after the demonstrated actions. Crucially, future observations and demonstrated actions are training inputs or targets only. During deployment, the predictor is removed. The selector uses current inputs, makes one hard selection per observation, and leaves the policy’s original action-generation procedure in place. “Future-aware” therefore describes its training supervision, not online access to future images.

Faster inference, with a pruning trade-off

On the LIBERO simulation benchmark, OpenVLA-OFT normally receives 512 visual tokens from two cameras. Keeping 128 reduced reported mean inference latency from 107.95 to 68.75 milliseconds, a 1.57-fold speedup. Average success across four task suites fell from 97% to 95.75%: a decrease of 1.25 percentage points.

Keeping only 64 tokens shortened latency slightly further, to 66.21 milliseconds, but success fell to 93.92%. In this experiment, more aggressive pruning therefore bought relatively little additional speed while losing more task performance.

The authors also tested π0.5 and XR-1 on RoboCasa365. Their reported APS-VLA results, averaged over two token budgets, show inference speedups of 1.37 and 1.52 times, respectively. Success remained close to the corresponding unpruned policies, although absolute performance varied substantially across models and task categories. Preserving a weak baseline is not the same as solving difficult tasks reliably.

A controlled LIBERO comparison supports the value of the future-feature objective: at 128 retained tokens, adding it raised development success from 94.58% to 95.75%. Another experiment found lower disagreement with the full-input policy when future targets matched the actual trajectory rather than being shuffled between episodes. These results support the training mechanism within the tested settings, not a universal guarantee.

What the physical trials establish

Across five tasks on two dual-arm platforms, each method received 40 trials per task. Mean policy inference latency fell from 137.25 to 93.25 milliseconds. Average success was 61% with APS-VLA and 60% without pruning. That small observed difference does not establish improved reliability; folding clothes succeeded in only 37.5% of trials for both methods.

The evidence also has comparison limits. LIBERO results use a development panel, and comparison timings include estimates and normalized reference values. The paper discloses reimplementing some comparison methods when official resources were unavailable, with language-model assistance. It promises code and checkpoints upon acceptance rather than documenting their release here.

This account uses the supplied preprint text extraction, which omits figures and may damage equations, tables or layout; no OCR was available. The findings are author-reported, not independently reproduced here. The practical promise is narrower but useful: learn a cheaper visual representation while preserving an existing policy, then measure carefully where that economy begins to cost successful actions.