Why AI discovery agents should compare competing explanations

An experiment can count against an idea without showing what is true instead. If an AI agent tests only one explanation against its complement, a rejection may tell it that the tested idea is unlikely—but not which of several alternatives the evidence supports. A paper by Ziang Liu and colleagues proposes a way to address this gap, called Contrastive Discovery.

Compare a set, not just a claim

The authors define a contrast set as a scientific question paired with explanations that are mutually exclusive and collectively exhaustive. That means the explanations do not overlap, and together they cover the possibilities under consideration.

In the pointwise approach described in the abstract, an agent tests an empirical claim against its complement. Contrastive Discovery instead scores an experiment by how its evidence redistributes belief across the contrast set. A question is considered resolved when belief concentrates on one explanation.

The authors say this changes the unit the agent proposes and scores, rather than requiring an entirely new discovery system. An existing agent could apply contrast sets to evidence it has already collected or use them to guide its search.

What the abstract reports

Across six domains, the authors report that 89% of the pointwise candidates with the largest belief shifts were rejections. Using the same evidence, contrast sets resolved 73% of those candidates to a specific alternative. The 73% refers to the candidates in that reported group, not necessarily to all questions the system considered.

When contrast sets guided search, the agent resolved 67% more questions than pointwise surprisal under the same experiment budget, according to the abstract. The abstract does not define pointwise surprisal in detail, so the comparison’s precise implementation cannot be assessed from the available account.

The authors also report a blinded study involving 38 human experts. Experts chose the same explanation as the system on 78% of resolved questions. On questions where a pointwise agent rejected its hypothesis and stopped, experts chose the same explanation as the contrastive system in 82% of cases. Agreement with experts is a comparison of judgments; by itself, it does not establish that an explanation is true.

What is still unclear

The source available for this article is the paper’s abstract, not its full text. It does not name the six domains or provide details about how questions and contrast sets were constructed, how belief was measured, or how the results varied by domain. Those details would help readers judge how broad the findings are and how the comparisons were carried out.

The method also depends on the explanations included in a contrast set. As a general caveat—not a reported finding of this abstract—if a set omits a plausible possibility or defines its options poorly, a choice among them may not settle the scientific question. The abstract does not tell us how the paper handles such cases.

The work is listed as an arXiv preprint. This article relies on the abstract and does not assess the full paper or its review status. The reported results suggest that explicitly comparing alternatives can help discovery agents extract more specific conclusions from evidence in the evaluated settings. How well that approach transfers to other questions remains unclear from the available source.

Source: Ziang Liu and colleagues, Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations, arXiv:2611.14514v1. This article is based on the abstract.