Better tools help AI run some materials simulations—but execution checks still matter

An AI-written scientific report presents two different questions: are its numbers close to a reference, and did the agent actually complete the required calculation? In DFT-Agent Bench, Shijie Liu and colleagues evaluate both. Their results suggest that specialized software support can improve particular materials-simulation workflows, without providing a universal advantage.

The study is an arXiv preprint, version 1 posted on November 7, 2026. Its conclusions concern two agent packages and two language models under a particular evaluation setup—not autonomous scientific research generally.

What the agents had to do

The benchmark uses Quantum ESPRESSO, a software package for density-functional-theory calculations of materials. Its released stable set contains 245 unique tasks involving simpler solids and more complex compounds. Additional exploratory tasks cover magnetic transition-metal materials.

Four workflows span increasing procedural complexity: relaxing a crystal’s structure; calculating electronic bands and density of states; fitting energy-versus-volume calculations to extract properties including resistance to compression; and calculating the elastic tensor, which describes responses to different deformations.

Reference answers came from the researchers’ converged Quantum ESPRESSO calculations, with public-reference cross-checks where available. Scores measure agreement with these references using task-specific tolerances. They are graded numerical scores, not percentages of tasks successfully solved.

The baseline, v1, is a relatively bare reasoning-and-action loop with generic file, shell, output-reading and submission tools. The richer v5.1 package adds structured simulation tools, input checks, domain guidance, a convergence sub-agent and verification. Both were evaluated with deepseek-v4-pro and claude-opus-4-8. Because these additions were bundled together, the comparison cannot identify which individual component caused an improvement.

Improvements depend on the workflow

Across matched tasks from the two stable material tiers, the richer package improved electronic-band and density-of-states scores by 0.08 for DeepSeek and 0.11 for Opus. The authors associate much of this improvement with a band-path protocol for metals that the baseline often missed. Exploratory magnetic-material relaxation tasks improved by 0.17 on both models.

Easier relaxation and equation-of-state tasks showed little improvement: baseline scores were already around 0.9 or higher. That leaves limited room within the scoring scale; it does not demonstrate a general ceiling on scientific capability.

Elastic calculations exposed a stronger model difference. DeepSeek’s score rose from 0.18 to 0.40. Its baseline repeatedly reversed a stress-sign convention in its fitting code. Opus’s baseline avoided that mistake and scored 0.62; the richer package scored 0.59, with no statistically significant improvement. Pooling the models would conceal these different outcomes.

A separate, fixed 100-task diagnostic disabled experience retrieval. With both DeepSeek packages rerun on frozen code, the mean score increased from 0.71 to 0.88. Opus’s richer package also reached 0.88, but its reported 0.10 gain used a historical baseline rather than an equivalently frozen rerun. A smaller matched retrieval comparison found no detectable benefit from enabling the experience store.

A good-looking answer can fail the execution contract

The benchmark separately checks whether Quantum ESPRESSO ran, whether the complete required calculation grid finished with completion markers, whether execution time was plausible, and whether network access was absent. Failing any check sets the score to zero.

In the main scored pool, this gate rejected 11 of 706 baseline runs and 15 of 942 richer-package runs—about 1.6% in each group. Twenty of these 26 runs otherwise scored at least 0.85. Some had incomplete calculation grids; six Opus relaxation runs from the richer package named the wrong output log despite perfect numerical scores.

These checks establish compliance with an execution contract, not dishonest intent. They also do not trace every submitted number all the way back to an audited output.

Why the conclusions remain narrow

The historical pool contains unequal repeat counts, changing prompts and evaluators, and best-of-many scores for some records. Repeated electronic-structure trials were noisy. The cleaner diagnostic was selected in advance but was not random, and its subgroup-selection information was not fully released.

Coverage of harder materials and longer workflows is incomplete. Public structures and reference data might also have appeared in model training. The appendix additionally leaves an invocation-budget question unresolved because the richer agent’s entry-point defaults differ from the stated comparison limits.

The defensible conclusion is therefore specific: specialized support helped selected evaluated workflows, while execution auditing caught failures that numerical agreement alone missed. Neither result establishes broadly reliable autonomous materials research.