RECAP reuses context-aware predictions to speed vision-language models

Vision-language models generate answers one token at a time. Speculative decoding tries to reduce that sequential work: propose several tokens, then have the original model check them together. A preprint by Hengjie Zhu and colleagues, submitted to arXiv on November 4, 2026, explores how to make those proposals from predictions the model has already computed, without training a separate drafting model.

Their method, RECAP, combines memories with different levels of context and different lifetimes. The reported results show higher average generation throughput than the tested training-free alternatives, while also documenting cases where individual inputs become slower.

Why the last token is not enough

Prediction recycling saves next-token predictions from earlier model evaluations and retrieves them to propose future text. The problem is deciding when an old prediction is relevant.

A memory indexed only by the last token can confuse different situations. The paper illustrates this with “her”: after “a red hat covers her,” a useful continuation might be “head”; after “blue shoes protect her,” it might be “feet.” An image adds another condition. The continuation of “the sign reads” depends on what the current picture actually shows.

The authors call this mismatch “condition aliasing.” Their image-replacement diagnostics connect it to draft rejection: the model’s first predicted token changed under image controls for 34.9% of fully rejected drafts, compared with 12.4% of drafts accepting at least five tokens. That is evidence of an association between image sensitivity and unsuccessful reuse, rather than a guarantee that contextual memory resolves every such case.

Remembering locally and across requests

RECAP stores prediction rows under keys containing one, two or three tokens. The longer keys distinguish short textual contexts that a one-token key merges.

Its two- and three-token memories reset for every request. Because the image remains fixed within a request, their predictions come from that request’s visual context. The one-token memory persists across requests within an inference stream, providing reusable transitions when more specific matches are unavailable. In the main experiments, this persistent memory was cleared between benchmarks.

Longer keys offer precision at the cost of availability. In an offline OCRBench replay, three-token keys matched only 18.6% of states, compared with 54.9% for one-token keys. Among matched states, however, they included the reference next token among eight candidates 90.6% of the time, versus 63.6% for one-token keys.

Rather than always choosing the longest match, RECAP mixes available predictions. It favors more specific keys and more concentrated probability distributions. This concentration-based confidence is not a direct measurement of agreement with the current image.

The combined predictions form a branching tree of candidate continuations. RECAP also inserts short continuations retrieved from matching suffixes in the prompt and accepted output history. The original model checks the tree in one packed pass, accepts the longest matching path and supplies an additional token. With exact target evaluations, this rule preserves the model’s greedy decoding output.

What the experiments show

The researchers tested Qwen2.5-VL-7B-Instruct and LLaVA-1.6 Vicuna-7B on an NVIDIA A100-SXM4-80GB GPU. Generation used greedy decoding, batch size one and a limit of 200 new tokens.

Across seven single-turn benchmarks, each containing 1,000 inputs, RECAP achieved mean throughput speedups over ordinary autoregressive decoding of 1.66× for Qwen and 1.56× for LLaVA. Token Recycling, the strongest baseline by mean speedup, reached 1.51× and 1.37×. These figures average paired input-level throughput ratios; they are not accuracy improvements.

On MMSpec, which includes multi-turn conversations, RECAP’s overall speedups were 1.89× and 1.59×. It did not win every category: on Qwen text-based visual question answering, Token Recycling reached 1.31× against RECAP’s 1.26×.

Component tests also qualify the headline result. Confidence weighting added modest mean throughput gains over fixed weighting: 0.73% on Qwen and 1.09% on LLaVA in four-dataset controls. Suffix insertion helped some datasets but reduced throughput on OCRBench. Average gains likewise did not protect every input: Qwen’s TextVQA results included a tenth-percentile speedup of 0.59×, meaning substantial slowdowns occurred within a benchmark that improved on average.

The boundaries of the claim

RECAP requires no auxiliary drafter parameters, but it does maintain prediction and sequence state in host memory. Comparisons with trained drafters used different autoregressive timing baselines, so their speedup factors do not establish that RECAP generates text faster in absolute terms.

The evidence covers image-based inputs on two 7B models under greedy, batch-one generation. Larger models, concurrent serving and video remain untested. The paper also does not establish improved downstream task accuracy or application-level latency distributions. Its contribution is narrower: better-organized reuse of existing predictions can accelerate the tested generation workloads, while the original model retains control over which tokens become output.