Structured Clinical Memory Shows Modest Gains in a Synthetic AI Decision Benchmark

An organized account of clinical context

A condition that is active, a past episode that has resolved, and a diagnosis that remains uncertain can have different implications for a clinical decision. Yet those distinctions may be scattered across records. In a preprint posted November 3, 2026, Jash Sayani, Gaurang Karwande, and Achinth Bharadwaj investigate whether explicitly organizing such information changes a model’s answers.

Their experiment uses Jev, a service that returns structured decisions and probabilities. The question is narrow: does adding structured clinical memory help when the model already receives the underlying patient vignette? The study does not establish whether the approach benefits clinical care.

What the researchers compared

The authors created 30 entirely synthetic cases across 12 clinical domains, including cardiology, diabetes, respiratory conditions, and mental health. No real patient records or protected health information were used. The cases contained relevant facts alongside ambiguity or distracting details.

Each case also received a synthetic memory block. Rather than a free-form summary, this was a list of condition or medication features. Fields recorded a normalized label, whether a feature was active, historical, or resolved, its onset, certainty, source form, and whether it was explicitly absent. An optional note supplied additional context.

The model answered the same questions under two conditions: the vignette alone, and that identical vignette followed by its memory block. Questions for each case were submitted together and evaluated in parallel, producing 60 calls across the two conditions.

The benchmark contained 56 multiple-choice questions, 36 binary questions, and 28 five-level ordinal scoring questions. Tasks included triage priority, next actions, condition presence, risk scoring, and contraindication checks. Both conditions used the same reference answers and rationales, which were single-author judgments without independent clinician review.

Correctness depended on question type. Multiple-choice answers had to match the reference option exactly. Binary answers used a 0.5 probability threshold. Ordinal answers counted as correct when their probability-weighted value was within one level of the reference score. Overall accuracy therefore combines different scoring rules rather than measuring exact agreement on every task.

Four additional correct answers

With the vignette alone, Jev answered 108 of 120 questions correctly, or 90.0%. Adding memory produced 112 correct answers, or 93.3%: an absolute increase of 3.3 percentage points.

Five answers changed from incorrect to correct, while one changed in the opposite direction. The paired statistical comparison, an exact two-sided McNemar test, yielded a p-value of 0.219. The result was not sufficient to reject equal paired error rates; it also does not establish that memory has no benefit.

The reported descriptive 95% confidence intervals overlapped: 84.6%–95.4% without memory and 88.9%–97.8% with it. These intervals have a further limitation: four questions shared each case, and treating questions as independent understates that clustering.

Accuracy increased descriptively across all three output types. Ordinal scores showed the largest increase, from 85.7% to 92.9%, but the small task-specific samples do not support firm conclusions about which tasks benefit most.

Probability scores improved too

The authors also measured the quality of probability outputs, not just whether answers crossed a correctness threshold. Multiple-choice log loss fell from 0.258 to 0.175, indicating greater probability assigned to reference options on average. Binary Brier score, which measures squared error in predicted probabilities, fell from 0.054 to 0.022. Lower values are better for both metrics.

Ordinal mean absolute error decreased from 0.521 to 0.469 levels. More predictions entered the highest-confidence group with memory, but sparse, uneven confidence bins and missing confidence fields prevent a claim of formally established calibration.

Memory increased average input length from 553.9 to 863.9 tokens, a 56% rise. Observed mean latency was similar—0.51 seconds without memory and 0.50 seconds with it. The small, networked run was not designed to establish a general latency guarantee.

A promising hypothesis, not clinical validation

The central limitation is the memory itself. It was synthetically supplied, not extracted by a production system. The authors describe it as an oracle upper bound: cleaner context that avoids errors such as stale status, incorrect negation, or merged conditions. Real extraction mistakes could reduce or reverse the observed benefit.

The experiment also cannot separate the value of structured formatting from the value of author-supplied normalization. It tested one model in one run, and the model revision was not recorded. No patient outcomes, safety endpoints, or demographic fairness analysis were assessed.

The authors propose larger, case-grouped evaluations using measured extraction output, independent clinician judgments, alternative context formats, and deliberately corrupted memory. For now, the findings support further testing of structured clinical context—not deployment claims or evidence of improved patient care.