When 4-Bit AI Models Run Faster—and When the Advantage Disappears

A practical question with fragmented evidence

Teams running open-weight language models need to know how much faster a compressed model will run and how much accuracy it might lose. This preprint examines that question for seven models in the roughly 9–20-billion-parameter range, using the serving engines vLLM and SGLang.

The authors reviewed documentation, model cards, benchmark reports and research papers through October 5, 2026. They retained timings that identified the engine, GPU, checkpoint and workload, and accuracy comparisons whose source supplied a BF16 baseline using the same evaluation harness. This is a synthesis of public measurements, not a new controlled benchmark.

Their central finding is an evidence gap: only two quantized checkpoints, both INT4 versions of Gemma 3 12B, had both serving timings and accuracy scores. Even those accuracy results covered only the philosophy subset of MMLU, a knowledge benchmark. The AWQ checkpoint had been calibrated using 30 examples from that same subset, compromising its independence as an evaluation.

Why smaller weights can accelerate generation

Quantization represents model values with fewer bits. BF16 uses 16 bits; weight-only INT4 recipes store weights in four bits but still perform matrix multiplications with 16-bit activations. FP8 recipes can reduce both weights and activations to eight bits.

The authors explain the available timings through the balance between moving data and performing calculations. With one sequence generating tokens, repeatedly reading model weights can be the bottleneck. Smaller weights reduce that traffic.

In one reported test, Qwen2.5-14B-Instruct on an A100 GPU with vLLM 0.6.3 generated 98.02 tokens per second using GPTQ-INT4, compared with 46.30 for BF16: a 2.12-fold speedup. That setting used a one-token input and 2,048 output tokens. With a 30,720-token input, the advantage fell to 1.64-fold.

Across the reviewed small-batch INT4 timings, speedups ranged from 1.35 to 2.12. They did not reach the fourfold reduction in weight size because attention, cached data and the work of converting compressed weights still impose costs.

Large batches change the bottleneck

Processing many sequences together lets a single weight read support more computation. Arithmetic can then become the limiting factor, reducing the advantage of weight-only compression.

For Gemma 3 12B on an H100, TorchAO INT4 reduced reported latency from 3.73 to 2.73 seconds at batch size one. At batch size 256, latency changed from 14.07 to 13.81 seconds—just a 1.02-fold speedup. AWQ showed a similarly small large-batch benefit.

The authors predict that FP8 could retain an advantage when computation dominates because supported hardware can perform lower-precision arithmetic faster. However, none of their qualifying sources tested FP8 above batch size one in this model range. That large-batch benefit is a prediction, not a demonstrated result here.

On an H20, one long-prompt Qwen3-14B test favored FP8 over AWQ: 1.68-fold versus 1.42-fold speedup. Its throughput counted both prompt and output tokens, so it cannot be compared directly with generation-only figures from other experiments.

Accuracy differences need careful interpretation

The accuracy tables are also heterogeneous. On full MMLU, reported eight-bit changes stayed within 0.77 percentage points of BF16. Weight-only INT4 losses ranged from 0.43 to 1.43 points; one NVFP4 result lost 1.63 points.

Some mathematics results were less reassuring. AWQ on Qwen3-14B lost 4.17 points on GSM8K, while GPTQ on Mistral NeMo lost 4.02. The authors identify these as the clearest losses relative to their estimated sampling-error yardstick.

No underlying source reported standard errors. The preprint estimates them assuming independent questions, but uncertainty in a difference also depends on which answers change between evaluations. Those paired results were unavailable. The authors therefore caution against using the reported HumanEval coding-score differences to rank recipes; this is not proof that the recipes are equivalent.

Evaluation settings further complicate comparisons. Two Qwen3-14B source groups had BF16 GSM8K baselines differing by 20.77 points, making comparisons across those groups unreliable.

What the evidence does not establish

The timing groups use different hardware, models and metrics, and the small-batch measurements predate current engine releases. The review found no qualifying AMD or Intel Gaudi serving timings in this range, nor FP4 speed comparisons against BF16. These are gaps in the reviewed record, not evidence that those platforms or formats cannot help.

The authors propose seven reporting requirements: exact checkpoint and revision; engine version and flags; GPU model and count; workload and throughput definition; a same-stack BF16 timing; evaluation settings with sample counts and standard errors; and calibration data separate from evaluation data.

The preprint supports a conditional takeaway: four-bit weights can substantially accelerate small-batch generation, but neither that speedup nor a particular accuracy cost should be assumed for another workload.