AutoEMatic adjusts AI number precision as a model writes

Making a language model cheaper to run often means representing its numbers with fewer bits. This process, called quantization, reduces storage and can accelerate calculations. But rounding numbers more aggressively can also change a model’s answers. The question is whether precision should remain fixed throughout a response—or change as the model works.

In an arXiv preprint marked under review, researchers from Georgia Tech and NVIDIA present AutoEMatic, a system that adjusts precision during inference. Their approach combines a decision policy, a numerical representation and implementations for both existing GPUs and a proposed custom accelerator. Its central aim is to recover accuracy without paying the full computational cost of higher precision everywhere.

Two signals for deciding when precision matters

The researchers first investigated how precision requirements vary across model layers and generation steps. For this analysis, they used an expensive reference procedure that compared quantized predictions with a full-precision model. It revealed changing requirements across prompts, layers and stages of a response. That reference is a research tool, not something AutoEMatic needs during ordinary operation.

Instead, the running system uses two internal signals. One measures how spread out the model’s next-token probabilities are. A spread-out distribution has higher entropy: several possible continuations are competing, so numerical changes may affect which token is selected. This is uncertainty about token selection, not a measure of factual correctness.

The other signal measures how much a layer changes its incoming hidden representation. Layers that make relatively small updates are treated as candidates for lower precision. Neither signal directly measures quantization error; together, they guide a practical rule-based policy.

During prompt processing, known as prefill, next-token entropy is not yet available. AutoEMatic starts at its highest supported precision and looks for a pronounced early spike in layer activity before allowing reductions. During token-by-token generation, or decode, it combines layer activity with entropy from the preceding step. Precision changes follow sustained trends across layers rather than every isolated fluctuation.

One stored base, several calculation modes

Changing precision would be cumbersome if it required storing several complete versions of a model. AutoEMatic’s AutoFP representation instead keeps an NVFP4 four-bit floating-point base and separate correction information.

Its higher-precision mode adds small signed corrections to selected groups of values. These corrections refine numbers that would otherwise suffer larger rounding errors. The mode is called AutoFP6, but it is not simply a conventional six-bit floating-point representation applied uniformly to every value.

The lowest mode, AutoFP2, maps weights onto five possible values, including zero. The name denotes a nominal precision level: the custom accelerator packs pairs of these values into five bits. Importantly, this lowest setting applies only to weights. Activations—the intermediate values calculated by the model—and the stored attention information called the key–value cache remain at four bits or above because reducing them further substantially harmed accuracy.

At the evaluated refinement budget, stored weights average 4.9 bits each, including scaling and correction overhead, compared with 4.5 for NVFP4. Weight preparation still uses offline calibration; runtime adaptation does not make the entire system calibration-free.

Accuracy recovery with a speed trade-off

The experiments covered reasoning, coding, long-context and agent-style benchmarks. On Qwen3.8-27B in τ²-bench, AutoEMatic scored 71.7%, compared with 50.0% for NVFP4—a gain of 21.7 percentage points—and matched the group-scaled six-bit baseline. On MiniMax-M2.7 in SWE-bench Verified, it scored 70.5%, versus 67.8% for NVFP4 and 71.0% for the eight-bit reference.

These are benchmark-specific results, not evidence of a universal improvement in model reliability. AutoEMatic often matched the six-bit baseline, but did not consistently recover the accuracy of the higher-precision reference.

On GB300 GPUs, the authors report approximately 90–98% of NVFP4 decode throughput across evaluated configurations. GPU kernels combine base calculations and corrections into a single enlarged matrix multiplication. However, the GPUs do not natively execute AutoFP2; that mode is emulated using four-bit hardware, so its potential computational advantage is not realized there.

Custom hardware results need separate interpretation

The proposed accelerator supports the numerical modes directly. Simulation and hardware-design estimates indicate 1.6–2.3-fold speedups and 7–23% lower energy than an NVFP4-only design when both reproduce AutoEMatic’s precision traces at the same accuracy. These are not measurements from a manufactured chip; off-chip memory timing also relies on an assumed bandwidth.

The comparison matters. In a separate experiment targeting NVFP4’s lower accuracy, AutoEMatic reduced latency by 12–14% but used 13% more energy. Nor does its lowest mode currently reduce memory traffic, because weights remain stored in the four-bit base. The demonstrated opportunity is selective numerical refinement—not a guarantee that fewer calculation bits always mean lower energy or smaller storage.