Four-bit weights can strengthen answer-slot preferences

A model might miss a multiple-choice question because it does not identify the right answer. It might also favor a letter—A, B, C or D—regardless of which answer appears there. Those behaviors are different: a letter preference is one kind of position effect, while order sensitivity more broadly concerns whether the selected answer changes when options move.

A new arXiv preprint examines how these effects relate to weight precision. The researchers tested Qwen3-0.6B and Qwen3-1.7B on ARC-Easy, ARC-Challenge, MMLU and HellaSwag, using 300 questions per benchmark. They compared a 16-bit floating-point reference with simulated round-to-nearest quantization at 8, 6, 4 and 3 bits. Quantization was applied to linear weights in the decoder blocks, in groups of 64. Embeddings, the tied output matrix and normalization weights remained in fp16.

The team read the models’ scores for answer letters after a fixed prompt. Each question was presented in eight arrangements: four cyclic shifts of the original options and four shifts of the reversed order. They measured accuracy, how often the selected answer changed across arrangements, and “slot skew”—how far the distribution of chosen letters differed from an even split. They also averaged option probabilities across up to four cyclic arrangements. To compare runs at different precisions and numbers of passes, they used a cost proxy based on weight bytes read, relative to one fp16 pass. This is not a runtime measurement.

At six bits, the models showed no accuracy loss against fp16 in these tests; their scores were modestly higher, a result the authors do not explain or claim as an improvement. Eight-bit scores were slightly lower. At four bits, accuracy fell by 14.3 percentage points for the 0.6B model and 5.6 points for the 1.7B model. At three bits, both were near the 25% chance level for four choices.

The clearest position-related change was in slot skew. At fp16, it was 0.15 for the 0.6B model and 0.07 for the 1.7B model. At four bits, it rose to 0.53 and 0.12, respectively. Thus, the smaller model’s four-bit version showed a much stronger preference for particular answer slots, alongside a larger accuracy decline.

That does not mean the study established a general increase in order sensitivity. Answer flips rose as precision fell, but an accuracy-matched baseline—one that estimates how many flips might accompany the same accuracy loss—predicted at least as many flips as the quantized models showed. This baseline is simple and assumes independent, uniformly random errors, so it is a limited comparison. The authors’ result is narrower: they detected a marked increase in slot skew beyond that baseline, but not excess answer flips.

A control added Gaussian noise to weights at the same signal-to-noise level as quantization, to test whether the effects were specific to rounding. The noise runs also showed substantial slot skew, and results varied between draws. The authors say this control is inconclusive: it does not establish that rounding itself causes the position preference.

Averaging probabilities across four cyclic orders improved accuracy, especially at four bits. Compared with one-order answers, the increase was 12.6 points for the 0.6B model and 4.9 points for the 1.7B model. These gains do not mean averaging restores all lost performance or removes every order effect.

Under the paper’s weight-traffic accounting, comparisons across the tested configurations generally favored keeping at least six bits and using any remaining budget for up to four orders. At very tight budgets, the authors’ decision procedure selected four-bit weights with one order. The rule depends on their cost definition and tested grid; it is not a universal recommendation for hardware or tasks.

The evidence has important limits. Quantization was simulated: the weights were not packed, and no low-bit inference kernels were run. The study covered two models from one family, one prompt format, four-option questions and 300 items per benchmark. Its cost proxy omits runtime factors such as activation traffic, dequantization and batching; counting the fp16 embedding and output matrix also changes estimated costs. The noise control used few draws, and the paper’s search for related work was limited to a title-level search of one index.

This is an arXiv preprint, not evidence of peer review. It offers a focused comparison in a specific setup: four-bit rounding coincided with stronger answer-slot preferences, while the study’s baseline did not show excess answer flips. Broader testing would be needed to learn how these results transfer to other quantizers, models, prompts, tasks and devices.