When rearranging the answers changes an AI’s choice

A test of consistency

Multiple-choice decision systems should distinguish answers by their meaning, not their place in a list. Mohammed Nayeem Khan and Logesh L, affiliated with Intel Corporation, examine that expectation in their preprint, Maya a1: Do the Options Read the Model, or Does Their Position?

Their subject is Laya, a model with approximately 421 million parameters. Unlike a chatbot that produces an answer token by token, Laya scores the available choices in one processing pass. The study asks whether moving those choices changes which underlying answer it selects.

How position enters the picture

In Laya’s default arrangement, the question and options form one sequence. Each option has a special marker, and the model uses its internal representation at that marker to calculate an option score.

The researchers inspected the released model and its software. They found that the final option-processing layers have no separate positional-embedding tensor, but the underlying text encoder uses position information. Moving an option therefore changes the context and position from which its representation is calculated.

Laya’s developers had already acknowledged positional effects and provided an alternative, parallel layout. This study’s contribution is measuring the problem on varied questions and comparing remedies, not discovering an entirely unrecognised issue.

What the researchers tested

The evaluation covered 951 choice questions with three, five or twelve options, including sentiment, support-ticket routing and hardware-failure classification. Qwen3.8-27B generated the examples. A separate call to the same model family checked their labels without seeing the intended answer; disagreements were discarded.

For three- and five-option questions, the researchers rotated the list so every option occupied every position. For twelve-option questions, they used six sampled arrangements, held constant across the models being compared. This produced 3,893 model calls per model version.

The main measure was the flip rate: the fraction of questions for which the selected answer changed across the tested arrangements. That measures consistency, not correctness. A model can choose the same wrong answer every time.

About one question in four changed answer

The released base model flipped on 25.8% of questions, with a reported 95% confidence interval of 22.9–28.7%. Among the 605 English-language items alone, the rate was 19.7%. The larger overall figure partly reflects difficult non-English examples.

Across the pooled presentations, accuracy was 77.1% when the designated correct option appeared first and 67.0% when it appeared last—a 10.1-percentage-point difference. This does not establish a universal first-option preference for every task.

Instability was associated with uncertainty. Dividing questions into three equal groups using the model’s average reported confidence across arrangements, flip rates were approximately 54%, 16% and 8%, from lowest to highest confidence. High confidence was therefore not a guarantee of consistency.

Training helped, but shuffling was only part of the gain

The researchers fine-tuned Laya on 2,818 teacher-checked training examples. For the version called Maya-v1, options were randomly reordered during training, with the target answer updated accordingly.

Its flip rate fell to 7.2%. However, training on exactly the same examples without shuffling also reduced the rate, to 11.4%. The matched comparison attributes an additional 4.2-percentage-point reduction to shuffling—not the entire improvement from the original model.

Accuracy averaged across arrangements rose from 69.1% for the base model to 86.3% for Maya-v1. The no-shuffle version reached 86.6%, so shuffling’s clearer advantage here was stability rather than higher accuracy. Repeated training runs supported the broad improvement over the base model, although results varied by seed.

Changing the layout went further

Adapting Laya’s existing parallel layout using the same training examples reduced the flip rate to 0.5%, with mean accuracy of 86.0%. This layout gives options shared starting position identifiers and prevents direct attention between different options.

Shuffled training did not eliminate every positional preference. When twelve options had identical descriptions, Maya-v1 still assigned its most-favoured position about 2.2 times the probability of its least-favoured position. The parallel version brought that ratio close to 1.2.

Repeated voting was less helpful: combining confidence-weighted answers across arrangements raised base-model accuracy only from 69.1% to 69.7%, at about 4.1 times the processing work.

What remains uncertain

These are synthetic, teacher-filtered tests, not human-labelled deployment records. An independent model family agreed with Qwen’s labels on roughly 94–99% of items in the reported checks, but agreement between AI systems does not establish human ground truth. Filtering can also remove difficult valid examples.

The headline audit samples only a small fraction of twelve-option arrangements. A larger audit found more unstable items, illustrating that measured flip rates depend on how extensively orderings are tested. Some examples also became consistently wrong after training.

The findings concern particular Laya versions and question formats, not all decision models. The supplied arXiv record identifies version 1, submitted on 6 November 2026; it does not report a peer-reviewed publication. The results support further order-sensitivity testing, not a claim of deployment-ready reliability.