When Clinician Edits Help—and When They Could Teach AI to Make Things Up

Clinical documentation tools can draft notes that clinicians then review and revise. Those revisions may offer examples for improving AI: they show how people changed a draft in a clinical workflow. But a changed note does not automatically reveal why a clinician made the change—or whether the added information was available to the model.

That distinction matters. If a model is trained to imitate an edit containing facts absent from its input, it may learn to add plausible-sounding information without evidence. The paper, Actionable Feedback: Improving Model Capabilities from User Edit Data on Clinical Documentation, investigates ways to filter edits toward those that are actionable and grounded in the model’s context. Grounding here means that information is supported by the input the model received; it does not by itself establish that the information is medically correct.

Sorting feedback by its relationship to the input

The authors say their review of data from a deployed clinical-documentation product found that more than 45% of edited outputs contained user-added information not directly supported by the model inputs. The abstract does not provide the sample size, review procedure or detailed breakdown. The figure should therefore be understood as the authors’ description of that review, not as a general estimate for all clinical AI tools or editing practices.

Because real clinician edits from deployed documentation tools are not publicly available, the team first developed a taxonomy of clinician-editing behaviors informed by data from such a product. They then used the taxonomy to generate semi-synthetic clinician-edit examples. These were constructed for the study, rather than being a public collection of actual edits from routine clinical care.

The task was generating discharge summaries from MIMIC-III, a public dataset. The researchers also tested simple filtering methods intended to identify actionable edits, evaluating filter accuracy through expert review. The abstract does not specify the filtering rules or explain in detail how reviewers assessed accuracy.

What the abstract reports

The authors report that fine-tuning on raw edited outputs increased ungrounded content in their semi-synthetic dataset. By contrast, progressively filtering the examples toward grounded edits improved clinician-alignment metrics while retaining faithfulness to the input context.

These results suggest that the edits used for training can matter: revisions that include information absent from the model’s input may have different consequences from revisions that stay grounded in that input. The study describes filtering as a way to select edits for training; it does not establish that retained edits are safe, medically correct, or proof that the resulting notes are better overall.

The abstract gives no numerical results for the alignment measures, filter accuracy or changes in ungrounded content. Without those details, readers cannot judge the size of the reported effects or the filters’ performance. The abstract also does not provide enough information to assess how well the approach might transfer beyond the study’s task and data.

A method demonstration, not a clinical deployment result

The reported experiment uses semi-synthetic edits and a discharge-summary task based on MIMIC-III. It does not establish that the method improves documentation in live clinical workflows, helps clinicians or patients, or prevents errors in deployed systems. Nor does the abstract show whether the findings generalize to other note types, institutions, models or patterns of clinician editing.

There is also a difference between being grounded in the model’s input and being medically correct. A statement can be supported by the supplied context yet still be wrong if that context is inaccurate or incomplete. The abstract reports retaining faithfulness to input context; it does not claim that this alone ensures clinical safety.

This is an arXiv preprint, and the evidence available here is its abstract rather than the full paper. The supplied source record says full-text extraction exceeded the request limit, and it does not establish the preprint’s review status. The findings are best treated as an early method demonstration. They raise a practical question for clinical AI training: before using human edits as supervision, how can a system identify changes that are grounded in the evidence it was given?