DARE looks beyond coding-agent scores to diagnose repair difficulty

A coding agent’s solve rate answers a simple question: how many software issues did it fix? It does not reveal whether those successes involved straightforward edits or demanding changes across a repository. Nor does it explain what help the agent might need next.

In the preprint DARE: Difficulty Analysis for Repair Enhancement, Jikwang Lee, Seungwook Jung and Jaechang Nam of Handong Global University propose connecting those questions. Their framework predicts repair-task difficulty, identifies which groups of task characteristics contribute most to that prediction, and uses the resulting profile to select additional support for a coding agent.

Defining difficulty through agent failures

The researchers define a task’s observed difficulty as the fraction of evaluated coding-agent configurations that fail it. Missing evaluations are excluded rather than counted as failures. This makes difficulty a property measured relative to an agent population, not an immutable attribute of a bug.

DARE predicts that measure using 151 static features drawn from issue descriptions, repository snapshots and history, reference patches, tests and task metadata. The features cover seven groups: specification, specification–solution gap, localization, transformation, validation, context and interactions between static features.

These groups describe different obstacles. Localization concerns identifying where changes belong; transformation concerns the extent and structure of the changes; validation concerns checking whether a repair works without breaking existing behavior. The specification–solution gap measures how directly an issue description relates to its solution artifacts.

A generalized additive model learns nonlinear relationships between individual features and difficulty while retaining separate feature contributions. DARE combines those contributions within each group. A task’s salient group is the one with the largest positive contribution after centering against training-task averages. These are explanations of the model’s prediction, not evidence that a particular characteristic caused an agent to fail.

Better at ranking than precise forecasting

The prediction study covers 2,594 tasks across four nonoverlapping partitions derived from SWE-bench and SWE-bench-Live Lite. Each evaluation trains on three partitions and holds out the fourth. Individual tasks do not overlap between training and evaluation, although some repositories do.

Because benchmark populations contain different agents, the researchers recalibrate predictions using outcomes from two shared anchor-agent configurations on the target partition. They then evaluate against the remaining agents. The procedure therefore requires some target-partition agent outcomes, rather than predicting an absolute failure scale entirely without target evaluations.

Reported Spearman rank correlations range from 0.476 to 0.621. The model correctly orders roughly 68–74% of task pairs with different observed difficulties. Absolute predictions are less precise: mean absolute errors range from 12.3 to 25.4 percentage points. DARE can therefore recover useful relative difficulty information without reliably assigning an exact failure probability to every task.

Turning a profile into assistance

The authors implement assistance modules for localization, transformation and validation. Localization support expands candidate locations and traces reproductions. Transformation support identifies dependent changes and checks remaining errors. Validation support examines coverage, runs relevant existing tests and checks potential gaps around modified code.

For the intervention experiment, the researchers first filter SWE-bench-Live Lite to 86 tasks meeting their issue–test consistency and reference-patch checks. They then select nine tasks predicted to be difficult: three for each supported salient group.

Using SWE-agent with Claude Sonnet 4.5, they run each task 25 times without assistance and 25 times with its corresponding module. Success means passing the benchmark evaluation tests. Across the selected tasks, successes rise from 123 of 225 baseline runs to 179 of 225 assisted runs: 54.7% to 79.6%. Every task improves, and the reported paired statistical test uses nine task-level observations.

What the experiment cannot settle

The comparison is matched assistance versus no assistance—not matched versus mismatched modules, random selection or providing all modules. It therefore does not isolate how much improvement comes from DARE’s selection strategy rather than from the modules themselves.

There is also an important deployment boundary. Reference-patch and test-derived features inform difficulty prediction and support selection, although reference patches and hidden tests are not supplied to the repair agent. This is a benchmark-informed setting, not a demonstration on newly reported bugs whose solutions are unknown.

The evidence is limited to Python tasks, one repair-agent configuration and nine selected intervention tasks. DARE offers a promising way to connect evaluation with practical support, but broader experiments are needed before its gains can be generalized.