CompressionBench tests whether coding agents can shrink language models without losing usefulness

A practical test of automated compression

Can an AI coding agent make an existing language model smaller while keeping it useful? In the preprint CompressionBench: How can your coding agents compress LLMs?, submitted on November 4, 2026, Xinrui Zhong and colleagues at Carnegie Mellon University investigate that question through constrained engineering tasks rather than proposed compression recipes.

The authors report that 29 of 36 agent submissions passed independent verification. Their results show that agents can combine numerical methods and inference code to produce working compressed models. They also reveal important qualifications: preserving a prediction score does not guarantee preserving generated answers, and smaller stored weights do not automatically make execution faster.

What the agents had to deliver

Each agent received a pretrained model checkpoint, a storage allowance, an offline software environment and eight hours on one H200 NVL GPU. Available resources included compression papers, calibration data and separate questions for local testing. Agents chose their methods, ran experiments and submitted both compressed weights and executable inference code.

Six agents each attempted six tasks. The source models were Nemotron-3.5 Super VL 120B-A12B and Qwen3.5 122B-A10B, both mixture-of-experts models, plus the dense Qwen3.8 27B. The larger models had allowances of 32 or 48 GiB; the dense model had 8 or 12 GiB.

Two storage measurements mattered independently: the saved package and the model weights retained on the GPU during inference. Persistent reconstructed weights counted against the allowance. Temporary workspace, attention caches and recurrent states did not. Consequently, passing the storage checks did not mean that total GPU memory consumption stayed below the same limit.

How success was measured

During development, agents received feedback based on perplexity, a measure of how well a model predicts scored tokens. Both the source and compressed models scored continuations generated by the source model. Lower compressed-model perplexity indicated closer preservation of the source’s predictive behavior on those histories.

Final verification used separate contexts and tested generated answers on GSM8K mathematics, HumanEval programming and TriviaQA factual questions. Each benchmark contributed fifty questions. Acceptance required the average accuracy across the three benchmarks to retain at least 80% of a source-model reference fixed before development. Execution, storage and integrity checks also had to pass. Decode speed was measured separately and did not affect the reward.

Strong results, with selection caveats

All eighteen submissions under the looser storage allowances passed. Under the tighter allowances, six failed the answer-accuracy requirement and one timed out during answer generation.

Selecting the lowest-perplexity agent artifact for each task produced an average perplexity increase of 3.4% over the BF16 source models. Those six selected artifacts averaged 77.33% answer accuracy, compared with 79.56% for freshly measured BF16 references. This is a best-per-task selection across agents, not the average performance of all submissions or the result of one agent succeeding equally well everywhere.

The recorded searches suggest that storage alone did not determine quality. In 21 of 24 revisions whose package sizes differed by no more than 0.01 GiB, agents reduced perplexity by refining representations or reallocating precision. In one controlled follow-up comparison, changing quantization clipping ranges improved prediction and answer scores without changing storage.

Why one score was insufficient

Cheap local measurements sometimes pointed agents toward the wrong candidate. Complete-model feedback reversed 17 of 79 prior local rankings. Some agents responded by restoring earlier candidates or changing their validation methods.

Even complete-model perplexity missed capability changes. In Opus’s tighter dense-model run, distillation and later refinements improved perplexity while mathematics accuracy fell from 88% to 72% across trained versions. Programming and factual-question performance improved over those same versions. Other compressed models produced repetitive or truncated programming answers, demonstrating why direct generation checks remained necessary.

Execution introduced another tradeoff. Sonnet’s 12 GiB dense artifact improved perplexity and answer accuracy over the released GSQ-RCO comparison model, but decoded at 16.7 tokens per second versus 74.6. Unpacking and reconstructing compressed weights can offset reduced weight storage. These measurements therefore do not establish automatic speed or energy savings.

What the preprint establishes—and leaves open

The main study used one run per agent–task pair and relatively small answer-test samples. Four additional repeated designs all passed, but they covered only two agents and two tasks; similar perplexity could still accompany different answer accuracy.

The authors frame compression as one measurable component of efficiency-oriented self-improvement, not a demonstration of recursive self-improvement. Whether the compressed models can themselves perform subsequent compression tasks remains untested. The evidence supports a narrower conclusion: agents can deliver useful compressed artifacts in this setting, but representation design, experimental judgment and validation of the final executable model all matter.