Medical AI scores fall sharply when answer choices disappear

Choosing the right diagnosis from a short list is different from producing it unaided. A new medical AI evaluation puts that distinction at the center of a broader question: what do impressive examination scores actually demonstrate?

Medical examination benchmarks offer a convenient way to compare language models. But selecting a supplied answer can draw on abilities that differ from generating one independently. In an October 2026 preprint, Navapat Nananukul and Mayank Kejriwal of the University of Southern California examine how much model performance depends on that distinction—and whether familiarity with public questions helps explain the scores.

The researchers evaluated 14 models from six providers on 3,076 questions across MedQA-USMLE, MedMCQA, PubMedQA and MedBullets. Their main comparisons used six smaller–larger model pairs; two additional open-weight models provided comparisons where parameter counts were disclosed. The evaluation covered medical licensing questions, entrance-examination material, biomedical abstracts and board-preparation cases. It was a benchmark study, not a test of patient care.

Strong scores, narrower implications

In the standard multiple-choice setting, every evaluated model exceeded the paper’s 60% MedQA-USMLE passing reference. The strongest models reached approximately 94.7%, above the 87% human-expert baseline drawn from earlier work.

That comparison did not generalize across datasets. The best PubMedQA result was 81.6%, but its confidence interval included the 78% expert reference. On MedMCQA, the best result, 83.8%, remained below the 90% expert baseline. The leading models also had overlapping confidence intervals, limiting what the study could establish about their individual rankings.

Removing the options changes the task

The clearest result came when the researchers withheld the answer choices. Across the 12-model main roster, mean accuracy fell from 82.3% to 50.9% on the 2,576 questions used for this comparison. PubMedQA was excluded because its answers remain yes, no or maybe even without a printed list.

Every main-roster model performed worse without choices. The two medically specialized MedGemma models showed the largest pooled declines: 40.0 percentage points for the 4B model and 37.6 points for the 27B version. GPT-5.6 Luna remained the strongest open-ended model, at 65.9%.

The authors interpret the gap as evidence that multiple-choice scores substantially reflect the ability to exploit supplied options. However, the comparison also changes the response instructions and scoring method: multiple-choice answers are checked against a letter key, while free-text answers are judged for substantive agreement with the reference. It therefore measures performance under two evaluation conditions, rather than isolating every possible influence of answer choices.

Reconstruction is not proof of recall

A third experiment hid the question and asked models to reconstruct its medical knowledge target from the choices, with a dataset-specific genre hint. Qwen 3.7 Plus achieved a 26.8% reconstruction rate; GPT-5.6 Luna reached 25.9%.

Those figures might suggest memorized examination content, but the supplementary controls complicate that interpretation. In a sample of 120 accepted reconstructions, model-based reviewers judged 107 inferable from the choices and textbook associations. Neither reviewer found a reconstruction reproducing a distinctive original detail. A separate guided-completion experiment found no clear evidence of verbatim memorization.

Models answered reconstructable questions more accurately, with average advantages of 7.4 percentage points in multiple choice and 14.5 points without choices. Yet question difficulty could explain both outcomes. A further analysis controlling for the individual question removed the association. The evidence therefore does not establish that memorization causes the higher scores—or that failed reconstruction proves a question was unseen.

Size, cost and grading matter

Across the six main provider pairs, larger models led by an average of 8.7 percentage points in multiple choice. On the three datasets used for open-ended evaluation, their lead widened from 10.1 to 13.4 points without options. These are not clean experiments on parameter count alone: some sizes were undisclosed, and the comparisons involved different model variants. The overall multiple-choice size result also did not survive the paper’s stricter correction for multiple statistical tests.

Smaller models often offered better accuracy per dollar, but the recorded prices were for hosted APIs, not hospital self-hosting. Combining models was no simple remedy: voting across all 12 reached 88.0%, versus 88.3% for the best individual model. A four-model committee selected with hindsight reached 89.4%, at 3.1 times the cost.

Free-text grading relied on Claude Sonnet 5. An independent licensed veterinarian agreed with 49 of 50 sampled answer verdicts, rating the remaining case borderline. That limited check supports the grading procedure without establishing comprehensive clinical validity.

The authors recommend refreshed questions, clinician-validated answer keys and more open-ended evaluation. Their central finding is narrower than a verdict on medical AI: strong performance at selecting an examination answer does not establish equally strong unaided answering, much less clinical competence.