VirtualTrialHarness reports gains in turning clinical protocols into executable cohorts

From a study protocol to a working cohort

A virtual trial, as defined in this paper, implements a protocol-defined treatment comparison using routinely collected electronic health records. Translating that protocol into code requires decisions that database schemas alone may not resolve: whether a medication order counts as treatment, which identifier represents a patient, and which timestamp anchors eligibility and follow-up.

In an arXiv preprint posted on November 3, 2026, Pengze Li and colleagues introduce VirtualTrialHarness, or VTHarness, to coordinate these decisions. The authors’ central claim concerns implementation reliability—not proof that an automated observational analysis produces valid treatment-effect estimates.

Procedures supplied at the relevant stage

VTHarness organizes work into five roles: protocol interpretation, source grounding, base-cohort construction, eligibility assessment and endpoint analysis. Although the architecture is described as multi-agent, the evaluated implementation used one language-model agent performing these roles sequentially.

Reusable “skills” describe local procedures, such as distinguishing medication administration from orders, linking patient identifiers and interpreting event times. The harness selects relevant skills and exposes sections appropriate to the current operation. Structured contracts pass accepted decisions between stages, including treatment groups, source choices and time anchors.

The workflow also supports execution checks and human review. A clarification can change an upstream decision and prompt regeneration of affected cohort outputs. These mechanisms are intended to keep later steps consistent with the study choices already accepted.

The strongest result comes from synthetic databases

The authors constructed VTBenchmark to evaluate source selection and cohort execution. Its execution component contains 24 base protocols, each represented in four synthetic database environments. These range from a relatively simple schema to a 1,024-table catalog with multi-step linkage paths.

Each task requires five linked products: patient–group–index records, condition-level flags, final eligibility, individual-condition counts and cumulative attrition. Complete delivery requires successful execution and reference agreement across all five products on both an original database and a version with relabeled identifiers.

The main experiment held the model, workflow, tools, clarification and execution budgets constant while varying knowledge support. Across the three complex environments, VTHarness delivered complete, reference-matched revised outputs in 137 of 144 workflows, or 95.1%. Query-driven retrieval from the same procedural corpus achieved 87 of 144, or 60.4%.

The reported difference was 34.7 percentage points, with a Bonferroni-adjusted 97.5% root-bootstrap confidence interval of 25.0 to 43.8 points. Neither schema-only support nor retrieval excluding procedural guidance completed a revised workflow in those complex environments.

This comparison supports the combined selection-and-exposure policy. It does not isolate whether content choice, ordering or context volume explains the improvement. Procedural retrieval and VTHarness also performed similarly in the simplest environment.

Additional evidence has narrower implications

A separate source-grounding evaluation used 90 constructed study tasks and three model backbones. Skills improved expected-table recall for GPT-5.1, Qwen3-30B-A3B and MedGo. Missing or invalid outputs received zero credit, so these results combine better output delivery with better source recovery.

For 13 historical Mayo Clinic requests, skills increased recall of tables appearing in analyst implementations from 26.6% to 60.7%. However, those notebooks reflected evolving physician requests. Greater overlap therefore indicates closer correspondence with existing code, not independent clinical validation. A separate author-adjudicated measure of requirement representation showed no consistent improvement.

After developing database-specific skills and execution procedures, the authors also obtained exact reference agreement on eight MIMIC-IV cohort tasks. Those same tasks and patient subset were used during adaptation, making this a development-set demonstration rather than evidence of held-out transfer.

Review remains essential

Revision was not uniformly beneficial: across all four synthetic environments, clarification repaired 19 VTHarness workflows but caused 11 previously correct workflows to become incorrect. Synthetic scoring also compared index dates rather than full timestamps, limiting what its agreement establishes about temporal precision.

Five experienced analysts estimated lower active workload with the system, with a median within-person reduction of 37.5%. These were scenario-based estimates assuming immediate physician clarification, not prospectively measured productivity gains.

The preprint offers evidence that coordinating local procedures can improve cohort implementation under the evaluated conditions. It does not establish autonomous clinical research or general reliability across hospitals. Public benchmark components and custom code were still awaiting institutional approval for release, and some institutional materials will remain restricted.