When execution traces help AI fix code—and when starting over works better

A wrong answer does not locate the mistake

An error message can tell a programmer where execution stopped. A failed assertion—a test checking whether a result meets an expected condition—often says only that the answer was wrong. The underlying mistake may be much earlier in the program.

In a preprint, Muntasir Adnan and Carlos C. N. Kuhn investigate whether execution traces can close that gap for large language models. These traces record intermediate variable values as code runs. Their findings suggest that traces help in some settings, but are not a general substitute for generating a new solution.

The researchers first examined failures remaining after an initial code generation and up to five repair attempts using structured error messages. Assertion failures represented about 79% of remaining failures on BigCodeBench, all remaining failures on HumanEval, and about 93% on LiveCodeBench.

Those percentages describe the failures left after repair, not all generated programs.

Showing the model what actually happened

The proposed method, Breakpoint Debugging, adds runtime values beside the lines of code that produced them. Assignments show variable values; loops show the values taken across iterations; control-flow annotations indicate where execution changed direction. The feedback also includes the failing input and the expected and actual outputs.

This replaces some of the model’s need to mentally simulate execution with a record it can inspect. Long sequences and values are shortened, and traces have size limits.

The method supplies traces only for assertion failures with visible test inputs. Other errors receive the usual structured feedback. A hidden test that reports only failure cannot provide a reproducible input for tracing.

The study used four sizes of Qwen2.5 Instruct models on Python tasks from three benchmarks: BigCodeBench, HumanEval and LiveCodeBench.

For tasks still ending in assertion failure after the baseline repair loop, the researchers reran generation and repair with either the first failing test’s trace or traces from all failing visible tests. They compared these with six independent fresh generations without repair feedback. The detailed protocol therefore compares restarted six-call strategies on selected failures, rather than simply adding five edits to the original failed program.

Where traces helped

On BigCodeBench, the main results table reports that one-trace debugging solved 299 of 1,014 selected task–model cases, or 29.5%, compared with 227, or 22.4%, for fresh sampling. Across the full benchmark, pooled accuracy reached 78.4% with one trace versus 76.8% with fresh sampling.

There is an internal reporting discrepancy: the abstract and conclusion give the one-trace repair rate as 26.7%, rather than the table’s 29.5%. The table and detailed results agree on the count of 299. That inconsistency warrants caution about the exact headline percentage.

On HumanEval, one trace solved 21 selected cases against 20 for fresh sampling. With only 40 selected cases, the authors say the comparison is too small to distinguish the approaches clearly.

On LiveCodeBench, fresh sampling solved 29 selected cases, compared with 22 for one trace. The reported difference was not statistically significant. Even when evaluated only on public tests, where traces were available, debugging showed no clear improvement.

More editing is not necessarily better reasoning

Traces substantially changed repair behavior. On BigCodeBench, unchanged-code resubmissions fell from 28% under baseline repair to 8% with one trace. Models also made larger edits, and more attempts increased the number of tests passed.

But those behavioral changes did not guarantee successful solutions. The authors suggest traces are most useful for implementation slips, such as an incorrect condition, rather than a mistaken algorithm or misunderstood requirement. A trace can accurately show a program following the wrong plan.

That explanation remains indirect: the researchers did not systematically label the underlying causes of failures.

Less trace can be more useful

One trace solved more selected cases overall than all traces on each benchmark. Although trace feedback lengthened individual calls, earlier success reduced total token use relative to baseline repair by 3.9% on BigCodeBench and 16.0% on HumanEval. On LiveCodeBench, total token use instead rose by 16.7%. Tokens measure model input and output volume, not directly elapsed time or monetary cost.

The evidence has limits. Each configuration ran once, so variation between repeated runs is unknown. The study covers one model family and function-level Python tasks, not large software projects.

Its practical lesson is conditional: a concise execution trace can help expose a local mistake, but fresh generation deserves an equal-budget comparison. More detailed feedback does not necessarily repair a solution built on the wrong idea.