When paying for both answers buys better evidence—and when it does not

A company commissioning expert checks on an AI benchmark wants more than a pile of verified findings. It wants information that helps it decide which tasks belong in the benchmark. Confirming what an expert already strongly suspects may be easier than uncovering a surprise, but less useful for the decision.

In Brave New Data: Pricing Expertise and Discovery, Dihan Zou of the University of North Carolina at Chapel Hill studies this tension using economic theory. The paper, submitted to arXiv on November 5, 2026, is a preprint; the supplied version lists no journal publication or peer-review status. Its results are mathematical predictions within a specified model, not findings from experiments with experts or purchasing data.

Two kinds of hidden information

The model has one buyer, one expert and two possible states of the world. The buyer must choose between two actions and wants to match the true state. A wrong choice carries a loss; payments to the expert are an additional expense.

The expert privately knows her initial probability assessment. She reports it, then chooses which state to investigate and how much effort to spend. Search can produce a certificate that conclusively establishes the targeted state. Greater effort makes that certificate more likely if the state is true. The expert can conceal evidence but cannot invent it.

Neither the initial knowledge nor the investigation is directly observable. The buyer therefore faces both a reporting problem and an effort problem: a contract must accommodate what the expert knows while encouraging useful additional work. The expert cares only about payment minus effort cost, not about the buyer's eventual action.

Zou first analyzes benchmarks in which the buyer can observe the initial assessment. There, investigating the less likely state is optimal: finding its certificate overturns the decision suggested by existing knowledge, while failure reinforces it. Hidden effort makes investigation more expensive to induce, but does not change that preferred direction.

Private assessments complicate this strategy. Experts can imitate another reported assessment and then choose their own search. A payment menu intended to reward surprising discoveries may instead let them collect attractive rewards for evidence they already expect to find.

Why unsuccessful search can be valuable

The paper proves that an optimal contract can use fixed rewards for disclosed certificates, independent of the initial report, and pay nothing when no certificate appears. The buyer still uses the report to interpret unsuccessful search and choose an action.

When investigation is sufficiently cheap, equal rewards for evidence of either state are optimal. Each expert searches for the state she considers more likely, maximizing her chance of earning a reward. That sounds like paying for confirmation rather than discovery. Yet intensive search makes failure informative too.

If a certificate should almost certainly appear when the favored state is true, not finding it can justify reversing the initial assessment. At the model's maximum effort, either the certificate or its absence reveals the state perfectly. Equal rewards thus accommodate the expert's preferred search direction without necessarily surrendering much accuracy.

A one-sided reward can cost less

As investigation becomes more costly, the optimal structure switches to rewarding evidence of only one state, despite the model treating both states symmetrically. Either state can be selected for the reward.

Experts who initially favor the unrewarded state now investigate the alternative. Their discoveries are harder to obtain, but change the buyer's decision whenever they occur. Experts who already favor the rewarded state continue to investigate it.

At this switch, the remaining reward rises, but paid discoveries become sufficiently less frequent that expected payments fall. The buyer accepts worse decision accuracy in exchange for those savings. The result is not that selective rewards produce better information; they can produce a better balance of information and expense for the buyer.

At still higher costs, the optimal reward drops abruptly. Some experts then investigate without improving the decision at all: their initial confidence is so strong that both a certificate and unsuccessful search lead to the same action. They still earn rewards because the common payment also motivates useful investigation by others.

Where the result stops

The characterization assumes uniformly distributed initial assessments, quadratic effort costs, nonnegative deterministic payments and perfectly revealing evidence. The binary states and actions are particularly important; richer decisions remain an extension. Experts with a stake in the outcome could also have different incentives to search or conceal findings.

Truthful reporting is only weakly optimal under report-independent rewards: experts are indifferent between reports, and the analysis selects truthful reporting. The contract does not make honesty uniquely attractive. These restrictions limit any direct purchasing prescription. The broader lesson is narrower: counting verified discoveries can miss whether the investigation improves decisions beyond what the expert already knew.