Can AI agents work with less context? A study tests compact memory and selective reading

An AI coding assistant can accumulate a lengthy record of commands, source files, errors and intermediate decisions. But does it need to reread that entire record before every action? In the arXiv preprint You don’t need long context, Peter Belcak and colleagues argue that, across their tested workloads, compact working memory and selective access to information can often substitute for large live contexts.

The qualification matters. Their findings do not establish that long context is useless or that every task can be solved equally well with a smaller window. They show that several benchmarks commonly associated with long contexts can be approached differently, with results that depend on the model and workflow.

Replace the transcript with a handoff

The researchers distinguish tasks that generate long histories from tasks supplied with large inputs. For the first group, they compare an agent that retains its complete conversation with one that periodically writes a compact handoff and starts a fresh conversation. The task and working environment remain available.

These handoffs are meant to preserve what matters next: the objective, established facts, changes already made, verification results, failed approaches and remaining work. The authors argue that a useful compaction is a brief for continued execution, not simply a narrative summary of everything that happened.

The principal accuracy experiments use Claude Opus 4.5 and GPT 5.6 Sol across six long-horizon benchmarks covering software engineering, terminal tasks, information retrieval and tool use. Many configurations restrict total context to between 16,000 and 48,000 tokens, with compaction triggered earlier to leave room for the handoff.

Results are often close to the full-history baseline, but not uniformly better. On SWE-bench Verified, Claude rises from 70.0% to 74.8% at a 24,000-token limit, then falls to 68.8% at the tightest 16,000-token setting. On FRAMES, a multi-hop question-answering benchmark, both models lose accuracy at every tested compaction setting.

Keep documents outside the working context

For large-input tasks, the researchers store documents as files rather than placing their full contents in the initial prompt. Agents search and read bounded passages using command-line tools. Searches can scan whole files, but only returned excerpts enter the model’s context.

Across seven long-input benchmarks, this approach generally matches or improves on full-input baselines. On AA-LCR, which tests reasoning across real-world documents, Claude reaches 84.0% accuracy with a 16,000-token context limit, compared with 65.0% for the full-input baseline. GPT reaches 83.0%, versus 82.0%.

These are results for a changed workflow, not evidence that compaction alone produced the gains. Several retrieval-oriented benchmarks require few or no compactions because selective reading keeps the conversation short from the outset. The comparison therefore includes the benefits of giving an agent tools and allowing it to choose which evidence to inspect.

One action, then a fresh conversation

The most aggressive experiment resets the conversation after each environment action and subsequent state update. Only a compact semantic state carries forward alongside the task, while the environment persists.

On SWE-bench Verified, this single-action regime scores 70.2% for Claude, against a 70.0% baseline, and 82.4% for GPT, against 84.4%. On Terminal-Bench, Claude improves while GPT declines. Median peak contexts shrink to roughly 7,000–9,000 tokens across these experiments.

The authors use this result to propose “semantic state space models”: models post-trained to act and maintain an explicit natural-language state between turns. That is a proposed direction, not a newly trained architecture demonstrated by the study.

Savings and failure cases

Separate serving measurements show why shorter contexts could be attractive. Under the paper’s cached-input, peak-throughput setup on H100 GPUs, reducing the configured window from 256,000 to 16,000 tokens increases aggregate decoding throughput by 3.6–7 times across three open-weight models. This measures output generation under specified serving conditions, not end-to-end task acceleration.

The paper’s larger task-cost savings are estimates that transfer a context-dependent pricing model from those measurements to the closed-weight experiments. They are not observed API discounts. Indeed, GPT’s nominal retail cost on SWE-bench Verified rises from $167.68 without compaction to $203.12 at the tightest setting.

Nor does every model tolerate aggressive compression. In complementary quantized open-weight experiments, Qwen 3 Coder Next falls from 53.0% to 3.6% on SWE-bench Verified at 16,000 tokens. The authors attribute much of that collapse to verbose trajectories leaving insufficient room for progress between compactions.

The practical lesson is narrower than the title: preserve access to information, but test how much must stay immediately visible. Compaction loses information; selective retrieval and persistent records can help recover it. Whether that trade works must be measured for the particular model and task.