A Small AI ‘Scout’ Could Help Predict the Cost of Larger Models

Reading the question, not generating an answer

In Scout Before You Route, Aristides Milios and Christopher Pal investigate a missing ingredient in model selection: the cost of answering a particular question, rather than an average question. Their paper was posted to arXiv on November 6, 2026. It is a preprint, so its results should be treated as preliminary rather than established findings.

A router chooses which language model should handle each request. It needs to balance the chance of a correct answer against the expected bill. But knowing a model’s token rates does not reveal how much output it will generate. In the study’s LiveCodeBench coding tests, output lengths between a candidate’s relatively short and relatively long problems differed by factors ranging from 11 to 121.

The researchers use a small, frozen language model as a “scout.” Frozen means its underlying weights are not updated. It processes the question once, without generating an answer. This initial processing stage, called prefill, produces internal numerical representations that become inputs to simple trained predictors, or probes.

For each candidate model, one probe estimates the probability of success and another predicts output length. The length estimate is converted into an expected cost using input and output token rates. The router then selects a candidate according to the value assigned to correctness relative to spending. Each question receives one answer, with no verifier or fallback attempt.

Testing whether length predictions pay

The main experiments used a four-billion-parameter Qwen3 scout and five candidates: two reasoning-effort settings each for gpt-oss-20b and gpt-oss-120b, plus deepseek-v4-flash. Nine benchmarks covered mathematics, coding, knowledge and mixed reasoning tasks.

The central comparison kept the success predictions unchanged while replacing the cost estimate. The reference router priced output using each candidate’s median training length. This isolates the contribution of predicting length rather than improving predictions of correctness.

At matched accuracy, the cost probes reduced spending significantly on eight benchmarks, from 9.4% on CodeContests to 33.0% on SuperGPQA. Savings were 27.8% on LiveCodeBench, 31.5% on Omni-MATH and 23.4% on MMLU-Pro. BigCodeBench showed no clear benefit.

These figures summarize comparisons across a shared range of accuracy levels, averaged over ten training-and-calibration splits. They are not a promise of the same discount at every operating point. A separate test that selected routing settings before seeing test results also found savings, although achieved accuracy differed from calibration targets.

An idealized comparison, giving the router advance knowledge of actual output lengths, saved 29–61% on eight benchmarks. That indicates substantial opportunity, but includes random output variation that a real predictor cannot foresee.

Difficulty is only part of the story

The researchers found that output length depended mainly on the problem: repeated answers to the same question varied less than answers across different questions, and different models often found the same questions lengthy.

On most mathematics and coding benchmarks, harder problems tended to produce longer outputs. Consequently, a length estimator built from predicted success performed nearly as well as the dedicated cost probe on several datasets.

That shortcut worked less well on knowledge and mixed-format tasks. A question can be difficult without requiring a long response, while its format can encourage lengthy output regardless of difficulty. On MMLU-Pro, the dedicated probe reduced spending by a further 18.4% relative to estimating length from success predictions.

In traffic mixing all nine benchmarks, the probe saved 22.4% beyond a baseline that already knew each question’s benchmark and used benchmark-specific median lengths. The benefit therefore went beyond recognizing broad task categories.

Where the approach falls short

BigCodeBench offered little measurable opportunity even with actual lengths known. CodeContests offered more opportunity, but all tested estimators captured relatively little of it. Savings were also smaller with non-reasoning candidates on the same problems.

With only 20 problems used to train the cost estimators, the probe retained 51–81% of its full-data savings across five tested benchmarks. Importantly, those experiments retained the existing success predictors; they do not show that an entire router can be trained from 20 examples. Transfer without target labels preserved substantial savings on four of seven new benchmarks, but was weaker elsewhere.

The scout also costs money. Its prefill pass is already needed by a success-predicting prefill router, but is an added expense compared with no routing. Larger scouts did not consistently improve savings beyond roughly two billion parameters.

Finally, the main pool covered only two model families, several comparison methods were reimplementations or adaptations, and statistical intervals were not corrected for multiple comparisons. Prices and provider behavior can change. The study supports predicting cost as well as correctness—not a universal claim that routing is cheaper, or that it matches the best model’s accuracy.