A Trial Simulator Can Look Accurate Without Knowing What the Drug Does

The missing control in trial prediction

An AI system forecasting a clinical trial can take shortcuts. The disease being studied, the outcome being measured and the choice of comparator may all help predict the reported result. None necessarily shows that the system knows what a particular treatment contributes.

In a November 2026 preprint, Alicem Koyun of Westcliff University proposes testing that distinction directly. Compare a model given intervention information with an otherwise equivalent version denied it. If identifying the treatment does not improve predictions, conventional accuracy scores alone offer weak evidence of treatment-specific capability.

The study evaluates computational forecasts against measured results from completed trials. It does not report a new clinical experiment, simulated patients receiving real treatments, or prospective evidence that AI can replace trial participants.

Hiding the treatment

Koyun began with 30,119 completed phase 2 or 3 interventional studies with results posted to ClinicalTrials.gov. After filtering and an arm-pairing audit, the benchmark contained 9,996 treatment-versus-comparator contrasts from 7,207 trials. These describe differences between groups on a primary outcome, expressed as risk differences or standardised differences for continuous measurements.

For the main binary-outcome evaluation, models learned from results posted before 2023 and were tested on 878 contrasts from 655 trials posted later. This chronological separation tests performance on more recent records rather than a random mixture of older and newer studies.

The intervention-blind reference retains information such as the condition, endpoint, eligibility criteria, population and comparator type, while withholding intervention-identifying fields. For language models, the comparison uses paired prompts rather than training a new model.

The paper’s “interventional skill” measures the fraction of the blind reference’s squared prediction error removed by supplying the intervention. A score of 0.20 means a 20% reduction in that error—not 20% greater patient benefit or 20% more successful trials.

Accuracy and treatment knowledge diverge

Koyun reports that the main registry-trained gradient-boosting model reduced squared error by about 21% relative to a simple prediction based on the training-set average. Yet its intervention-blind counterpart performed better. Interventional skill was approximately −0.12, with a 95% confidence interval from −0.21 to −0.05.

Alternative representations and analyses did not establish a convincing positive contribution from intervention information. The author interprets this as no detectable additional treatment-specific skill in the registry-trained systems tested, not proof that every possible registry-based model must fail.

The largest language model tested, identified in the paper as Claude Sonnet 5, performed differently. Using registered protocol fields without posted results-group descriptions, it achieved interventional skill of 0.20, with a 95% interval of 0.08–0.31. Two smaller models produced estimates near 0.08, with intervals reaching or crossing zero.

Additional computational perturbations supported the large model’s sensitivity to treatment information. Swapping the arms reversed its prediction’s sign in 86% of contrasts with non-trivial original predictions. Substituting an unrelated drug performed worse than withholding the intervention. These checks used prompts containing posted results-group descriptions, however, rather than the cleaner protocol-only format.

What the comparison cannot establish

The blind reference is not entirely blind. Disease and eligibility details can imply which treatment is being tested. In a separate probe, a language model recovered the withheld drug class from blind prompts 45% of the time, versus roughly 2% under a randomised comparison. Koyun therefore treats interventional skill as an incremental contribution beyond those remaining clues.

Memorisation is another unresolved issue. The study could not construct a useful post-training-cutoff evaluation for the largest model. For an open-weight model, a subset of 217 contrasts meeting the study’s post-cutoff criteria yielded protocol-only skill of 0.01, with an interval spanning −0.10 to 0.12. That result was too imprecise to settle the exposure question and does not resolve it for the largest model.

The benchmark also represents a selected population of trials with posted, usable results. It reduces varied endpoints to two effect families and uses aggregate rather than individual-patient information. A fresh pairing audit found 96 of 100 sampled contrasts valid, leaving residual construction error. Positive interventional skill could still exploit correlations rather than treatment mechanisms.

A benchmark, not permission to replace trials

Uncertainty estimates add a practical warning. The large model’s stated 90% prediction intervals covered about 70% of observed contrasts. For the registry model, error-control thresholds calibrated on earlier data did not maintain their targets under temporal shift.

The preprint’s strongest contribution is therefore an evaluation control: require models to demonstrate what intervention information adds, rather than treating overall accuracy as sufficient. It does not establish readiness to shrink or replace clinical-trial control arms.

This account relies on the supplied extracted text and explicit prose results; damaged table or formula layouts and unavailable figure imagery limit independent interpretation of visual details.