ServerBench tests whether coding agents can build better LLM serving systems

Testing the whole serving system

Can coding agents improve the infrastructure that runs language models? ServerBench, a preprint by Xinrui Zhong and colleagues at Carnegie Mellon University and the University of Southern California, turns that question into an engineering benchmark. Version 1 was submitted to arXiv on November 4, 2026.

Rather than asking agents to optimize an isolated calculation or patch an existing engine, the benchmark asks them to build a complete, specialized serving system. Each task fixes the model checkpoint, hardware, workload and inference requirements. Agents start from a minimal Transformers server and receive eight hours to develop an implementation and choose how many application sessions it handles simultaneously.

That operating choice matters. More sessions can increase useful work, but excessive load can create queues and slow responses. ServerBench therefore evaluates software design and load selection together.

Value counts only when responses arrive on time

The benchmark’s score, called “dollars per dollar,” divides latency-qualified token value by allocated GPU cost. Input and output tokens receive fixed prices, but a completed request contributes value only if it meets both response-time targets.

Those targets cover time to first token—the wait before a response starts—and time per output token, which measures generation speed. The authors calibrate the thresholds using SGLang reference replays and freeze them before development. Agents then choose their own session concurrency.

This is an efficiency measure, not demonstrated profit. It excludes development expenditure and other operating costs. Its accounting also values cached input prefixes at the full input-token price once per logical request. The score therefore reflects the benchmark’s specified pricing rules rather than a universal business outcome.

Replaying dependent sessions

ServerBench uses recorded coding-agent sessions through the AgentX harness. Requests retain varying lengths, shared context, dependent turns and parallel subagent branches. Later requests wait for the responses they depend on, so server speed affects how much work finishes during the measurement window.

For privacy, reconstruction replaces original prompt and tool contents while preserving workload structure. The resulting tasks reproduce structural serving demands, not the original coding problems.

Agents work without internet access in isolated environments. They can experiment with batching, scheduling, memory management and computation reuse while preserving the prescribed model behavior. Final evaluation rebuilds the frozen submission in a fresh deployment and runs a 1,800-second replay using seeds separate from development.

Strong gains, uneven delivery

The study evaluates 11 coding-agent models across three deployments: Gemma E4B on one L40 GPU, Qwen3.5 9B on two L40s, and Qwen3.8 27B INT4 on two L40s.

The authors report verified systems in 22 of 33 agent–deployment entries, with 13 exceeding the strongest baseline for their deployment. Claude Sonnet 5.5 scores 2.97 dollars per dollar on Gemma E4B, 4.87 times the vLLM reference. Claude Opus 5.5 leads the other two deployments, scoring 4.40 on Qwen3.5 9B and 2.64 on Qwen3.8 27B INT4.

These are gains in the benchmark’s qualified-value score, not equivalent claims about raw generation speed. Only three evaluated agent models exceed the strongest reference on all three deployments.

What helped—and what failed

The authors identify three recurring lessons. First, successful designs coordinate capacity improvements with load. In one Gemma trajectory, Sonnet 5.5 improves its engine before raising concurrency, then selects 32 sessions rather than 36 because the higher setting causes more generation-latency misses.

Second, experiments need workload-relevant inputs. One agent rejects prompt-lookup speculation after a random-text probe shows little benefit. Another investigates actual replay replies and adopts it. In a controlled continuation experiment, providing workload feedback leads an agent to re-adopt speculation; its submitted system delivers 1.13–1.31 times the qualified value on two fresh common seeds.

Third, verification must exercise the paths optimization changes. Short tests miss an admission policy that stalls on a permitted 315,000-token prompt and a cache defect triggered by repeated prefix insertion. Conversely, deliberately forcing cache preemption and restoration exposes another system’s invalid outputs and enables repairs. Final checks examine workload execution, generation fidelity, answer accuracy and prohibited shortcuts through behavioral tests and code review.

What the results establish

The evidence remains limited to three deployments and one coding-trace source. Development and evaluation sample the same corpus, and closed-loop replay differs from open-loop production demand. Short measurement windows also overestimate longer-window scores in the authors’ comparisons.

The study generally allows one development run per agent–deployment pair, with a possible rerun for clearly anomalous failure. The authors caution that repeated runs are needed to separate agent variability from workload variability, and that verification has finite coverage. ServerBench demonstrates promising measured artifacts—not reliable autonomous delivery across arbitrary serving environments.