FrontierPhysics tests whether AI agents can complete physics research—not just solve problems

A benchmark built around research projects

In the preprint FrontierPhysics: Evaluating Agents for End-to-End Physics Research, submitted on November 4, 2026, Bingran You and colleagues introduce a benchmark intended to move beyond fixed physics questions and isolated scientific subtasks. The study evaluates research workflows, not simply whether an agent can produce a correct final number.

The benchmark contains 56 tasks developed by 42 physics researchers from their own research projects. These were selected from 182 submissions through general quality checks, domain-expert review and inspection of trial agent runs. That review concerns the benchmark tasks; it should not be confused with journal peer review of the preprint.

The tasks span nine domains, including condensed matter physics, astrophysics, fluid dynamics and quantum information. They cover theoretical, computational and experimental research modes. Agents receive a high-level brief and any supporting materials needed for the task, but no explicit solution pathway. They must investigate the topic, develop an approach, conduct the work and deliver both verifiable outputs and a scientific manuscript.

Correct numbers are not enough

FrontierPhysics combines two kinds of evaluation. Deterministic tests check structured outputs against reference values. A separate reviewer agent examines the manuscript, execution history, code and other files using expert-authored criteria.

Some criteria are mandatory “blockers,” representing requirements essential to scientific validity. An attempt passes only if it clears every deterministic test and every blocker. Other criteria assess qualities such as literature synthesis, reproducibility and presentation, contributing to a weighted quality score. Strong writing cannot compensate for an incorrect numerical result or a failed scientific requirement.

The authors describe this framework as assessing whether work is correct and publishable. Here, publishability is a benchmark judgment under its tests and rubrics, not evidence that an agent’s manuscript has been accepted by a journal.

One example asks an agent to study a quantum transition using a simulated Rydberg-atom apparatus. The agent must characterize the device, construct a model, simulate larger chains and estimate the transition point with uncertainty. This requires connecting measurements, physical approximations, numerical methods and literature—not merely applying a supplied formula.

The strongest configuration passed about one attempt in four

The experiments cover 22 models in 26 configurations across four agent harnesses—the software environments through which models use tools. Each configuration has three designated attempts on each of the 56 tasks, giving 168 evaluation slots, with a uniform ten-hour timeout per attempt.

Claude Opus 5.5 in Claude Code achieved the highest observed overall pass rate: 27.4%, or 46 of 168 attempts. GPT-6 Astra at maximum inference effort followed at 23.2%, while GPT-6.1 Sol achieved 19.6%.

These are configuration-level results, not comparisons of models operating in an identical software environment. Nor does the numerical ordering establish a decisive lead: the paper’s task-level statistical analysis did not distinguish Opus 5.5 from several nearby configurations at its stated significance threshold.

The gap between numerical correctness and overall success was substantial. Opus 5.5 passed all deterministic checks in 45.8% of attempts, compared with its 27.4% overall pass rate. Across three attempts, it passed at least once on 25 of the 56 tasks.

Where research workflows broke down

Experts manually examined 168 GPT-6 Astra attempts. Among 143 runs classified as failures in the recorded assessment, lack of rigorous thinking was tagged in 48%, missing expert intuition in 34%, literature mistakes in 27% and instruction-following problems in 26%. Multiple tags could apply to one run.

Examples included circular self-verification: an agent checked its results using a method that depended on the same mistaken assumption as its original calculation. Other agents neglected edge cases or kept adjusting parameters without identifying the underlying problem. Literature failures included missing important prior work or failing to use references to support scientific claims.

Wrong physics was tagged in 16% and execution mistakes in 4%. The authors therefore interpret reasoning rigor and expert intuition as more prominent weaknesses in this audited cohort. They explicitly caution that other models and model–harness combinations may fail differently.

The graders also needed scrutiny

The audit found nine attempts whose failure outcomes should change because of task-design or grading problems. Issues included ambiguous instructions, overly strict tolerances and reviewer mistakes. The authors report fixing task defects and rerunning or regrading affected results.

Evaluation uncertainty remains important. A second reviewer agent was more lenient toward outputs from its own model family, and the small numbers of tasks within individual physics domains make domain comparisons uncertain.

The preprint thus presents two connected findings: completing these research workflows remains difficult for the tested agents, and measuring that ability requires expert scrutiny of the benchmark itself. The authors argue that future evaluations should better accommodate exploratory reasoning, partial progress and unexpected findings.