TRANSMISSION // 2611.23174v1
When coding agents search for a finding, ordinary statistical tests can mislead
Source published
Coding agents can test many analysis choices and report the most promising result. A study argues that this selection can inflate false positives—and proposes replaying the agent’s analysis on subsamples to adjust for it.

AI coding agents can do more than run a statistical test: they can explore data and try different ways to analyze it. That flexibility may make an apparently significant result less convincing than its p-value suggests. A new preprint measures the risk in simulations and proposes a correction based on replaying the agent’s own choices.
Coding agents can now write analysis code, manipulate data and run statistical tests. That could make empirical research faster—but it also raises a question: if an agent tries many plausible ways to analyze a dataset and then presents a compelling result, does a standard significance test still mean what researchers think it means?
In Inference after coding agents, Florian Gunsilius, David Van Dijcke and Kaspar Wüthrich argue that this workflow can intensify a familiar statistical problem: selective inference. When researchers choose an analysis after seeing which results look promising, the reported test may understate the chance of a false alarm. The authors’ concern is that coding agents can search broadly across analysis choices, then replay their selected approach in a reproducible way. They describe this as a new form of post-selection inference—not because selection is new, but because the search and the analysis can both be carried out programmatically.
A simulation with no real effects
The authors tested four frontier coding agents on simulated data in which the true effects were zero. They asked the agents to produce a compelling finding. In this setting, the abstract reports that the agents produced significant p-values between 54% and 98% of the time.
That range is a warning about the particular experiment, not a general estimate of how often coding agents produce false findings in real research. The task explicitly asked for a compelling result, and the test used simulated data with no effects. The result demonstrates how selection can make conventional p-values misleading under those conditions; it does not establish that agents routinely produce false positives across scientific fields or ordinary research tasks.
Learning from the agent’s choices
The researchers propose a way to account for this selection. After an agent completes an analysis, they reset and replay it on subsamples of the data. The replays are used to estimate how likely the agent is to select that analysis when the result looks statistically favorable. The correction therefore learns from the agent’s observed choices rather than requiring researchers to write down a complete model of how the agent searches.
In simulations, the authors report that this procedure reduced the high false-rejection rates to near the nominal 5% level. They also state a theoretical condition: for agents whose selection behavior is stable, the corrected test is asymptotically valid and consistent conditional on the agent having selected that analysis. In accessible terms, the guarantee is not claimed for every possible agent or search process; it depends on the selection behavior meeting the stated stability condition.
The authors also illustrate the method with applications drawn from two published papers. The abstract does not provide enough detail to describe those applications or their results, so they should not be taken here as evidence of a particular substantive conclusion.
What remains uncertain
The proposed correction is an attempt to address a specific problem: an agent’s search for a striking result can affect the reliability of the test applied to that result. It does not, on the evidence reported in the abstract, establish that every form of agent-assisted research is biased, or that replay solves other concerns such as poor measurement, flawed data or inappropriate research questions.
The findings are reported in an arXiv preprint, not a peer-reviewed publication in the source information provided. This article is based on the paper’s abstract because the full paper text was not available in the supplied material. Details needed to assess the experiments—such as the agents’ identities, simulation design and assumptions behind the correction—are therefore not independently examined here. The results should be read as an early methodological argument and simulation finding, not a final verdict on how AI agents affect empirical science.
The practical takeaway is cautious: when an analysis is selected after an agent has explored many alternatives, a conventional p-value may not account for that search. Replaying the agent’s selection process offers one proposed way to measure and correct the distortion, but its reliability depends on the method’s assumptions and on further scrutiny of the full study.
- coding agents
- statistical inference
- selective inference
- p-values
- AI in science
- false positives
Related reading
Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.
When AI text detectors should say “not enough text”
Source published
Shared keywords: false positives
PHISHREV+ uses explicit rules to reconsider phishing alerts
Source published
Shared keywords: false positives
When a Spillover Test Decides What Researchers Test Next
Source published
Shared keywords: selective inference
Estimating Long-Term Effects Across the Outcome Distribution
Source published
Shared keywords: statistical inference
When a Few Large Groups Distort Statistical Inference
Source published
Shared keywords: statistical inference