When Should a Clinical AI Decide, Ask, or Defer?

When a medical case is missing information, “don’t decide yet” can mean two different things. Perhaps one more patient-specific clue could help. Or perhaps the case should be handed to a clinician. Many evaluations collapse both possibilities into a single “abstain” option, making it hard to tell whether a model recognizes a chance to gather evidence or simply avoids making a diagnosis.

Mingzhan Yang and Weili Wu propose evaluating three actions instead: DECIDE, by giving an autonomous diagnosis; ASK, by selecting one additional piece of case information; or DEFER, by handing the case to a clinician. Their study asks whether this distinction reveals behavior hidden by a binary decide-or-abstain test—and whether the additional information a model selects improves its eventual diagnosis.

Testing choices with information held back

The researchers evaluated Qwen, Gemini and GPT on 50 synthetic diagnostic cases from DDXPlus. Each case was presented at four evidence levels—25%, 50%, 75% and 100% of its available evidence—for 200 matched evidence states. The models saw the clinical information available at each stage but were not told how complete it was or what information remained hidden.

If a model chose ASK, it selected one question from a case-specific pool of unrevealed evidence. The answer came from the original case record, rather than being generated by another model. After seeing that answer, the model had to decide or defer; it could not ask again. The researchers also compared Qwen’s selected question with one randomly selected from the same pool, and compared its three-action policy with a decide-or-abstain baseline using the same evidence states.

This setup makes the choices comparable, but it is not an open-ended clinical conversation: the model could choose only among existing clues. Nor does the benchmark establish which action was clinically best for each case.

Models often chose different paths

The models disagreed about the initial action in 137 of the 200 states. Their overall patterns differed: Qwen selected ASK more often, while Gemini selected DECIDE more often and GPT selected DEFER more often than the other two. At full evidence, models still disagreed in 37 of 50 cases. These results show differences in action selection—not that one model was safer or better calibrated.

Qwen’s selected questions changed what happened next more clearly than whether the diagnosis was correct. In 105 matched comparisons, the model chose to decide after targeted information in 67.62% of cases, versus 57.14% after random information. The study reports this difference as inconclusive: the confidence interval and statistical test gave slightly different signals near the conventional significance threshold. Correct diagnoses increased from 38.10% to 43.81%, but that difference was also inconclusive. Among cases where Qwen did decide, its accuracy was similar in the targeted and random conditions. The authors therefore interpret the result cautiously: selecting information may affect willingness to make an autonomous decision without reliably improving correctness.

The binary comparison also showed a tradeoff, not a clear overall win. Qwen’s three-action policy reduced autonomous coverage—the share of states ending in a diagnosis—by 14.5 percentage points and correct autonomous coverage by 13.5 points. The overall wrong-decision rate fell by 1 point, a difference that was not statistically significant. Some wrong diagnoses under the binary baseline became either correct diagnoses or deferrals under the new policy; some correct diagnoses were no longer made autonomously. A deferral is counted as a non-decision here, not as a proven clinically appropriate choice.

What the study can—and cannot—show

The action menu makes an important distinction visible: seeking more information and sending a case to a clinician are different behaviors. But adding an ASK option did not produce a uniformly better decision policy. The results suggest that knowing when to gather information, choosing useful information and knowing when evidence supports a diagnosis are separate challenges.

The evidence is limited. The evaluation used 50 synthetic cases, not real clinical workflows or real patient data. ASK was restricted to one clue from a finite pool, and the targeted-versus-random and binary-baseline analyses were conducted only for Qwen. Only one random comparator was drawn per ASK, and model-generation settings were not fully standardized across providers. Most importantly, DDXPlus labels diagnoses but does not say whether DECIDE, ASK or DEFER was the right action in each state.

The work is a preprint presented as a NeurIPS 2026 workshop paper, not evidence of a validated clinical system. Its findings describe model behavior in a controlled benchmark; they do not establish clinical safety or readiness for patient care. Testing action choices in more realistic settings, with clinically grounded criteria and human oversight, remains necessary.