For AI Research Agents, Naming an Experiment Is Not Designing One

The gap between a suggestion and an experiment

An instruction to investigate a model component can sound scientifically sensible without specifying an executable experiment. Someone—or another software system—must still choose the interventions and number of repetitions. Those choices affect both cost and what the results could reveal.

In the research preprint Designing Informative Actions: Action Construction for Long-Horizon Research Agents, Alkut Arxidin and colleagues propose evaluating that gap explicitly. Their evidence concerns simulated experiments and judgments of constructed research proposals, not demonstrated improvements in scientific discovery.

The central question is practical: how much useful experimental detail does an agent actually communicate, rather than leave its executor to supply?

Scoring the possible interpretations

The authors introduce a “reading set”: the executable designs that a fixed interpretation policy assigns to an instruction. An incomplete request can produce several designs. A valid instruction specifying the interventions and repetition count identifies just one.

Each design receives an expected information-gain score, representing how much its possible outcomes could reduce uncertainty among competing explanations. The evaluator divides that information by an abstract experimental cost. Designs exceeding the budget contribute zero, but remain in the average across interpretations. This prevents affordable alternatives from hiding an unaffordable interpretation.

The resulting average is compared with the best affordable information-per-cost value available in the benchmark. A score of 100% therefore means matching that benchmark optimum, not achieving perfect scientific understanding. Costs are dimensionless resource units, not measured computing time, and exclude model-call charges and generation tokens.

What complete instructions changed

The controlled benchmark contains 30 synthetic research states involving Transformer components, partial results, competing mechanisms and budgets. Each state permits 693 combinations of interventions and repetition counts.

The study compares four interfaces: an action name with a reason; a target with a justification; a complete design specifying target, interventions, repetitions and success criterion; and a complete design with an added prediction. Two model families, DeepSeek and Qwen3.8-27B, were tested under two request settings each.

Averaging settings within each family, the action-name interface achieved 13.36% of optimal affordable efficiency. Complete-design requests reached 75.59% for DeepSeek and 57.15% for Qwen3.8-27B. Those changes correspond to the reported relative increases of 465.98% and 327.91%. They are improvements in the simulation’s normalized score, not percentage-point gains or measured increases in research productivity.

Among 720 main complete-design and design-with-prediction requests, 715 produced a parsed design within budget. Failures remained in the scoring denominator.

The interface matters as much as the wording

An important qualification is built into the comparison: the action-name parser extracts only the action name, even when the accompanying explanation mentions experimental details. Its fixed completion policy consequently gives all tested model settings identical action-name scores, state by state.

The experiment therefore evaluates the combined request-and-parser interface. It does not isolate whether models possess better scientific reasoning when asked for more fields.

Controls reinforce that distinction. Short, complete instructions preserved the observed gain, while longer action-name explanations did not improve scores when their details were ignored by the parser. Adding predictions produced no statistically significant additional gain after correction for multiple comparisons. That does not establish that predictions are useless or that the conditions are equivalent.

Agreement with research judgments

A separate author-constructed dataset contains 60 scenarios and 240 candidate actions across machine learning, systems and AI for science. The scenarios draw on 29 source papers, but their priors, budgets and simulator mappings are evaluation assumptions. Proposed follow-up experiments have no newly measured outcomes.

Eighteen participants reporting graduate research experience supplied pairwise judgments. The scenario-level information-per-cost measure had a mean within-scenario Spearman correlation of 0.733 with aggregated human rankings. A six-model panel also showed positive agreement.

These correlations indicate agreement about proposals, not proof that the proposed experiments would resolve hypotheses. Humans judged displayed instructions, whereas the metric also incorporated executor completion. The paper reports cases where those perspectives produced different preferences.

What remains unproven

The synthetic results depend on a finite observation model, a restricted design space, an abstract cost rule and declared completion policies. Complete specification removes ambiguity under those policies, but does not guarantee a good design.

The constructed scenarios can share source papers, and independent human validation of every scenario is not established. Participant experience was self-reported; twelve sessions finished in under ten minutes and were retained after review.

Despite the title’s long-horizon framing, the reported evaluations do not demonstrate improved end-to-end research campaigns. Executing the proposed experiments and measuring actual hypothesis resolution remain future work. The preprint’s contribution is narrower: making the design information delivered by an agent—and the consequential choices left to its executor—measurable.