A New Way to Shrink What AI Agents Read from Their Tools

The problem with a growing search trail

A search-enabled AI agent may need several queries to answer one question. Each query returns snippets or documents, and the agent typically adds them to the material it consults on its next step. That accumulation can become lengthy even when only a small part of each result is useful.

In a preprint, Yuetong Liu and colleagues at Tsinghua University propose a different way to pass tool results to an agent. Their method, called Coca-Cola, compresses an observation into latent tokens: vectors of numbers that the model can process, rather than words a person can read. The aim is to retain what matters for the agent’s next action without keeping the full search output in its working context.

The distinction matters. Existing approaches can remove selected words or ask another language model to write a shorter summary. Coca-Cola instead inserts numerical representations directly into the target agent’s input embedding space—the internal format in which the model receives token information.

Compressing for a decision

The compressor takes three inputs: the task instruction, the agent’s current reasoning state and the latest tool observation. A small language model encodes them, and an adapter converts its output into a fixed number of latent tokens compatible with the agent. At inference, the process repeats after each tool call; the latent tokens replace the raw observation in the agent’s subsequent context.

The researchers train the compressor to support decisions rather than to reproduce the original text. They run the same target agent with a full observation and with a compressed one, then encourage the compressed version’s predicted next-action distribution to match the full version’s. A second training objective uses the next action recorded in the training trajectory. The target agent’s parameters remain frozen while the compressor is updated.

This is an important distinction from simply preserving every detail. For a question about a racing winner’s birth year, for example, the winner’s name could guide the next search, while many details of the race may not be needed for that decision. The method is trained to favor such action-relevant information, although the representation itself is not a readable summary.

What the experiments found

The authors evaluated web-search agents on four question-answering benchmarks: Natural Questions and TriviaQA, which primarily test retrieval of factual evidence, and 2WikiMultiHopQA and HotpotQA, which require connecting multiple pieces of evidence. They used two model configurations, one from the Qwen family and one from the GLM family. The default setup compressed each observation into 128 latent tokens, with up to five search results per call and five calls per task.

The comparison included an uncompressed agent, a token-pruning method, a language-model summarizer and an adapted latent compressor trained to reconstruct the original observation. Answer quality was measured using exact match and token-level F1, which captures overlap with the reference answer.

Relative to the uncompressed agent, Coca-Cola reduced combined input and output token consumption by 70.01%–89.24% and reported end-to-end runtime by 24.45%–55.97% across the tested settings. It also improved both answer metrics over the uncompressed agent in all eight model-and-dataset combinations. For example, on TriviaQA with the Qwen configuration, F1 rose from 62.87% to 67.07%.

It was not the top scorer against every compression baseline. On Qwen TriviaQA, the reconstruction-based latent baseline achieved higher exact match and F1; on Qwen 2Wiki, token pruning achieved higher F1. Coca-Cola nevertheless had the best scores on both metrics in six of the eight model-and-dataset combinations.

What the results do—and do not—show

An additional test on Natural Questions with the GLM configuration found a trade-off in the number of latent tokens: F1 rose as the budget increased from 8 to 128, while the highest budget produced a lower overall compression rate. Removing the task goal or reasoning state from the compressor’s inputs also reduced exact-match performance in that test. These findings support conditioning compression on what the agent is trying to do, rather than on the observation alone.

The authors suggest that filtering irrelevant information may help explain the answer-quality gains, but the experiments do not establish that explanation as the cause. Their illustrative case also does not isolate compression as the only difference: the compressed and uncompressed runs received differently ranked search results.

The evidence is confined to the reported web-search question-answering settings. Although the architecture is not specific to search, the paper does not test it on other tools. Its runtime measurements also depend on the described hardware arrangement: the target agent ran on one GPU, and where an additional GPU was used, it hosted the compression model. The preprint therefore makes a case for a new interface between tools and agents, not a demonstration that the same savings or accuracy gains will hold for every agent workflow.