How do we judge human–AI teamwork when there is no single right answer?

Better together—according to what standard?

An AI assistant improves a clinician’s performance. Does that mean the partnership succeeds? Not necessarily: the clinician working with AI might still perform worse than the AI alone. Researchers use complementarity for the stronger condition in which the partnership outperforms both participants acting separately.

Measuring this usually requires a common reference answer. In a diagnostic task, researchers might compare a clinician, an AI and their combined judgment against disease labels established through tests and expert assessment. But what is the reference answer when the task is choosing a treatment rather than identifying a disease?

In the theoretical preprint Complementarity without ground truth?, Andrea Ferrario, Alessandro Facchini and Federico Cabitza propose extending this comparison to practical decisions. Their contribution is a framework for evaluation, not a clinical experiment demonstrating better treatment choices or patient outcomes.

A treatment is not a diagnosis

The authors distinguish a condition in the world from the procedure used to establish a reference label. Even diagnostic labels require choices: which observations count, which instruments are used and how experts resolve disagreement. These procedures aim to establish what is the case, although some diagnostic targets are substantially more contested than others.

Treatment selection adds another layer. Clinical evidence about benefits and harms matters, but so do patient preferences, treatment burden, available resources and acceptable trade-offs. Knowing all the descriptive facts would not automatically settle how these considerations should be balanced.

There is also a problem of missing alternatives. Once a patient receives one treatment, clinicians cannot directly observe how that same patient would have fared under every other option. A successful outcome therefore does not, by itself, establish that the chosen treatment was the best available.

The proposal is not to abandon evidence. It is to distinguish a reference intended to describe reality from one intended to judge whether an action is adequate for a specified purpose.

Building an adequacy benchmark

The framework constructs what the authors call adequacy labels in two steps, drawing on multi-criteria decision analysis: methods for comparing alternatives across several dimensions.

First, each available treatment is assessed separately against relevant criteria, using patient information and the decision context. Second, an explicit scoring procedure combines those assessments so that the options can be compared. The highest-scoring treatment becomes the reference label, with rules for resolving ties specified and justified.

An illustrative emergency-department example considers three levels of respiratory support: conventional oxygen, advanced non-invasive support, and invasive ventilation with intensive-care escalation. Assessment dimensions include expected benefit, deterioration risk, treatment burden and feasibility. Patient measurements inform those assessments; staffing, equipment and intensive-care capacity help define the context.

This is an example of the framework’s structure, not a validated treatment-selection tool. The paper does not supply a clinically tested scoring recipe for these choices.

Once labels exist, researchers can score the human, AI and partnership against the same reference. Complementarity means the partnership has a lower average penalty than either participant alone. The comparison remains familiar; what changes is how the benchmark is constructed.

Transparency is not enough

Documenting a scoring rule explains how it produced an answer. It does not establish that the answer is defensible.

The authors therefore identify supporting evidence inspired by computational reliabilism, an approach to justifying reliance on computational results. Relevant checks include agreement between independent assessors, clinical evidence supporting individual criteria, patient participation and replication across institutions. That support remains provisional and specific to the task and setting.

They also propose a robustness test. If several scoring rules are independently defensible, the partnership should outperform both stand-alone participants under every rule in the specified set—not merely under the most favorable one.

What a positive score would—and would not—show

Compressing several considerations into one label can conceal important trade-offs. Some decisions may be better represented by several acceptable actions or an incomplete ranking, possibilities this framework does not fully develop.

A positive result would establish better performance relative to the documented adequacy benchmark. It would not establish a universally best treatment, improved patient outcomes or overall system reliability. The proposal’s central move is to make the values behind “better” explicit enough to examine, challenge and test.