A debugger did not beat a second shell-based AI repair attempt

A live debugger lets a programmer pause running software, inspect variables and step through instructions. An AI coding agent working through a shell can instead edit files, insert print statements and repeatedly run tests. Both routes expose evidence about a bug, but do they help agents solve different problems?

In an arXiv preprint, Ryan Magnuson, Ellie Kitanidis and Ariel Lee of Code Metal compared the two approaches. The landing page lists acceptance at the NeurIPS 2026 workshop “Who Verifies the Agents? Toward Reliable Agent Development.” Their main finding is narrower than a verdict against debuggers: for the two models and repair environment tested, replacing shell access with a debugger reduced success, and mixing the two interfaces did not beat repeating the stronger approach.

Comparing repair attempts

The researchers used all 500 tasks in SWE-bench Verified, a human-validated collection of software issues. A successful repair had to fix the designated failing tests without breaking designated passing tests. Known problematic tasks remained in the evaluation and counted as failures.

They tested GPT-OSS-120B and Nemotron-3-Ultra-550B. Each model used two agent configurations, with three independent attempts per configuration on every task. The agents shared navigation, testing and editing tools, prompts and other experimental settings. One configuration could execute arbitrary shell commands; the other used a persistent Python debugger and restricted Python execution for setup.

There was an important procedural difference beyond tool availability. The debugger agent could not edit until a debugger command succeeded in the current work segment. The researchers introduced this gate after pilot runs showed repeated edits without runtime inspection. The shell agent received an instruction to observe before editing, but no equivalent enforced gate.

Each attempt allowed up to ten work segments, called relays, containing up to 50 agent turns apiece. At a boundary, a summary carried progress into a fresh model context while file changes persisted. Equal relay limits did not mean equal token consumption.

Why a unique success can mislead

An agent’s solved-task list changes between attempts. A debugger agent might fix a task that one shell attempt misses, even though another shell attempt would also fix it. Comparing just one run of each can therefore make ordinary sampling variation look like specialized ability.

The paper introduces a statistic called cross-pass-at-k to estimate the chance that at least one attempt succeeds when attempts come from different agent configurations. It combines repeated observations rather than relying on a single pairing. Its unbiasedness depends on independent, identically distributed runs within each configuration and independence between configurations.

At the full budget, both models’ shell agents averaged 56.7% success per attempt. Debugger-agent success averaged 49.6% for Nemotron and 40.3% for GPT-OSS.

For Nemotron, two shell attempts had an estimated combined success rate of 62.7%, compared with 60.3% for one shell attempt plus one debugger attempt. For GPT-OSS, the corresponding figures were 64.7% and 60.0%. Two shell attempts led at every relay threshold examined. Pooling repeated runs also reduced the apparent debugger-exclusive coverage, supporting the authors’ interpretation that much of it reflected run-to-run variation.

Adding a debugger is a different question

A separate experiment retained the shell and added debugger access, without the editing gate. Average single-attempt success was close to the shell-only configuration: 58.1% for Nemotron and 57.9% for GPT-OSS.

Pairing a shell-only attempt with a shell-plus-debugger attempt reached 65.6% and 66.3%, respectively, at the full budget—above two shell-only attempts. The authors interpret these results as near-equivalent configurations rather than convincing evidence of distinct complementary abilities. The substitution result should not be read as showing that every debugger-equipped combination performs worse.

What remains uncertain

The study covers two models, one debugger and a Python-only benchmark. Tasks involving persistent, state-dependent failures may yield different results, and the largest-patch groups were too small for strong conclusions.

Training exposure is another uncertainty. The authors note that Nemotron’s technical report lists SWE-bench Verified among agentic post-training seed sets without specifying the split. Agents could also read the benchmark’s failing tests during attempts, unlike leaderboard-style evaluation.

The practical lesson is about comparison: before crediting a new tool with unique capabilities, check whether another attempt with the existing tool achieves more.