AI agents propose useful experiments, but choosing the best remains harder

An AI research assistant might suggest five experiments, implement them correctly and still struggle with a more consequential question: which one should run first? In the rTaste preprint, Art Kanke of the University of Minnesota investigates this early-stage research judgment, calling it research “taste.” The findings suggest that proposing worthwhile experiments and selecting among them are distinct capabilities.

Testing every proposal

The study gave six models seven recent natural-language-processing papers with released code. Each model worked both as a single researcher and as a team of instances of the same model. The team used two independent proposers, a critic and a lead responsible for selecting and revising the final proposals.

Each condition was asked to register five minimum viable experiments: inexpensive tests intended to indicate whether a direction deserves further investment. Before execution, the proposing agent assigned a probability to each experiment meeting a threshold it had specified, then ranked the five by how worth pursuing they seemed.

Crucially, the ranking did not determine which experiments ran. Separate executor models attempted every proposal under common resource limits. This let the study examine low-ranked ideas rather than leaving their potential value unknown. Of 84 planned conditions, 83 produced proposals, yielding 415 experiments; 412 were ultimately delivered.

The experiments were small by design, restricted to open models of at most 8 billion parameters and short single-GPU runs. Submitted evaluations were rerun on fresh machines, but those reruns reused the executor’s code. Three model auditors therefore separately assessed whether implementations followed their registrations, produced valid evidence and had research value.

Useful does not necessarily mean unexpected

The study classified 194 experiments as sound, valuable and new. “Sound” meant faithful to the registration and valid; “valuable” meant receiving one of the two highest research-value ratings; “new” meant neither of two literature-searching novelty judges found equivalent or closely related prior work.

Those definitions matter. A follow-up can be unpublished yet obvious to researchers reading the same paper. Among the 194 experiments meeting all three criteria, only 73 also satisfied a separate model-review criterion for going beyond the paper in a non-obvious way.

Across delivered experiments, reviewers classified 66% as building on the seed paper, 24% as checking its claims and 5% as opening a new direction. GPT-6 Astra and GPT-6.1 Sol proposed about two-thirds of the experiments rated both beyond the paper and non-obvious. These are judgments within this evaluation, not independently established measures of scientific originality.

Rankings showed limited added value

Agents selected the auditors’ most-valued experiment in 38% of conditions, compared with 20% for a random choice. However, their rankings were not shown to improve on simply running the experiment they had written first. The estimated advantage over that baseline was three percentage points, with a 95% confidence interval ranging from minus eight to plus fifteen points.

That uncertainty does not establish that ranking is useless. It means this study did not demonstrate an advantage over the simpler baseline. Agents chose their first-written experiment as their top priority in 47% of conditions, suggesting substantial alignment between writing order and later ranking.

Forecasting showed a similar distinction between capability and added value. Agents predicted threshold outcomes more accurately than a base-rate forecast derived from the other seed papers. Yet their forecasts were not measurably better than those from a fresh instance of the same model given only the seed paper and registrations, without the proposer’s search history or reasoning.

Passing a threshold is not the same as learning

An experiment’s registered event depended on the bar its proposer chose. Some models typically required a confidence-interval bound to cross a threshold; others usually compared a point estimate. Consequently, event rates were not interchangeable measures of research quality.

Reviewers judged half of the sound experiments whose registered events did not occur to provide informative evidence, mostly against their hypotheses. Conversely, they judged 20% of sound experiments whose events occurred inconclusive. Meeting a target and answering a scientific question were therefore different outcomes.

What the evidence leaves open

Teams produced an estimated 0.71 more sound, valuable and new experiments per five proposals than single researchers, but the confidence interval included zero. Average costs, including proposal generation, implementation and reruns, were $23.93 per team condition versus $15.31 for a single researcher.

The study remains a preprint with important limits: seven seed papers, one run per condition, no human baseline and model-generated evaluation labels. Two auditors also participated as studied models and executors. Rankings carried no actual budget consequences, so behavior could differ when prioritization determines what runs.

Kanke proposes testing that next: give agents a fixed budget, let them revise decisions after initial results and compare their allocations with human researchers. The present evidence supports useful experimental assistance more clearly than reliable autonomous research prioritization.