The same AI model can ace a test—or fail when cut off

An AI benchmark score looks like a verdict on a model. But between receiving a question and being awarded a mark, the model passes through a chain of software decisions. How long may it respond? Which number counts as its answer? How does the hardware calculate its next word?

In a book-length arXiv preprint, Luv Verma reports experiments showing that these decisions can change scores, generated text and serving performance without changing the model’s trained weights. The central argument is practical: a result describes a model running under particular conditions, not the model in isolation.

When the test stops the answer

One experiment evaluated Qwen3-8B on all 1,319 questions in GSM8K, a benchmark of mathematics word problems. Using the vLLM 0.11.0 inference engine, Verma held the prompts, model, decoding and scoring rules fixed while changing the maximum response length.

At eight tokens—small units of text—strict accuracy was 0.08%. Every response reached the limit. At 1,024 tokens, accuracy reached 93.18%, and only 0.2% of responses were cut off.

This was not a newly trained model becoming better at mathematics. The shorter setting generally prevented it from reaching its final answer. The experiment illustrates a measurement failure, not a universal benefit from longer reasoning.

A separate response-limit sweep makes that distinction clearer. Beyond a 1,024-token limit, accuracy stayed approximately flat. Raising the limit to 4,096 increased average actual output from 290.4 to 293.3 tokens. Four times the permitted length did not mean four times the text generated—or establish four times the cost.

The marking software matters too

Allowing an answer to finish does not settle how it should be graded. Another experiment generated a set of Qwen3-8B mathematics responses once, then applied three rules for extracting answers from the identical text.

Taking the last number produced 90.37% accuracy. Requiring a number after a particular marker produced 87.87%. A phrase-based heuristic, intended to recognize answer statements, produced 54.59%.

The 35.78-percentage-point spread arose after generation. Some rules missed final answers; another sometimes selected intermediate arithmetic. None was shown to be universally best. The lesson is that the answer extractor belongs in the benchmark specification, alongside the questions and model.

Different wording is not necessarily worse performance

The preprint also examines why disabling random sampling does not guarantee identical output. Greedy decoding chooses the highest-scoring next token, but different numerical implementations can still change which token wins.

Switching between FlashAttention and FlashInfer attention backends changed generated text on roughly 39–55% of prompts across three tested model families. Small numerical differences offer a plausible explanation, but the experiments did not trace the precise internal operation responsible.

More importantly, changed text and changed correctness are separate measurements. In another study, Verma compared prefix caching on and off. This optimization reuses previously computed information for shared beginnings of requests.

Across five disjoint 200-question mathematics slices, three model setups showed about 34–47% text divergence. Mean accuracy changes were small relative to variation between slices. Yet approximately 1.1–2.0% of all paired items changed from correct with caching off to wrong with caching on. Improvements elsewhere largely balanced those losses.

An almost unchanged average therefore did not mean that nothing consequential changed. Nor does this cache-specific finding establish that other execution changes are harmless.

What the evidence can establish

The same caution applies to long context. In a synthetic Qwen3-8B test, retrieving planted strings remained perfect through approximately 28,000 tokens, while six-step variable tracking scored only 9 out of 30 at approximately 16,000 tokens. Finding information and following dependent updates were different capabilities.

These are reported findings from a preprint and should be treated as provisional. Conditions, sample sizes and repetition vary across experiments. The volume does not distribute executable code, raw generations, complete run records or immutable model revisions, limiting independent verification and exact reproduction. It also discloses unresolved or inconsistent cells, including an omitted context-tracking result.

The useful takeaway is consequently a reporting discipline rather than a new leaderboard: record the exact model, runtime, prompts, response limits, scoring rules and workload. Then test the property that matters—correct answers, identical text, latency or cost—instead of assuming one measurement establishes the others.