A tougher coding benchmark finds leading AI agents still struggle to finish the job

Short requests, substantial changes

ExpertCrowd-SWE contains 100 tasks across 11 programming languages, drawing on 87 open-source projects. Experienced engineers wrote requests for codebases they knew, rather than collecting already-published fixes. The median request contains 148 words; the median reference solution changes 471 lines across several files.

The tasks ask agents to navigate repositories, understand existing conventions and coordinate changes across components. Examples include database migrations, authentication behaviour and compiler functionality. Authors were forbidden to base tasks on bugs already fixed publicly, and their reference solutions had to remain unpublished.

Each agent worked in an isolated software environment with dependencies already available and no network access. Hidden grading materials were inaccessible to it. The researchers evaluated seven models, usually with five scored attempts per task, producing 3,495 scored runs.

Measuring more than whether tests pass

The benchmark’s central choice is to award partial credit. Each task has weighted requirements, mostly checked by executing software through its public interfaces. An agent can therefore receive credit for working parts of a change without being counted as having completed the whole request.

Two additional rules prevent missing essentials from disappearing inside a respectable average. A failed “blocker” caps the score. A failed “gate”—such as changing existing tests when that is forbidden—sets the score to zero. Fourteen tasks use gates.

The authors checked every grader against a reference solution, an untouched repository and a comment-only change. These had to receive full credit, zero and zero respectively. Some tasks also included alternative correct solutions or deliberately inadequate patches to challenge the grader. Those checks provide evidence of grading quality, although they cannot establish that every possible valid or invalid solution will be classified correctly.

Ten tasks include a small amount of language-model judging. Across the benchmark, those criteria account for 0.85% of the total available weight; the other 90 tasks use entirely deterministic checks.

Two leaders, but no decisive winner

Claude Opus 5.5 achieved an average score of 67.3%, followed by GPT-6 Astra at 63.7%. Their 3.6-percentage-point difference was not statistically significant in the researchers’ paired comparison. The estimated interval for that difference extended from a 1.4-point disadvantage to a 9.7-point advantage for Opus. That leaves the ordering uncertain rather than demonstrating identical ability.

Claude Fable 5.1 scored 46.5%, followed by Gemini 3.8 Flash at 40.0%. Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 scored between 32.2% and 33.5%, with no statistically clear separation within that group.

These percentages are average rubric scores, not percentages of tasks completely solved. Opus earned full credit on 26.6% of its scored runs, and Astra on 23.4%. Every other model’s full-credit rate was 8.6% or lower.

The leaders also differed in repeatability. Opus completed more tasks at least once across five attempts: 47, against Astra’s 36. Astra achieved full credit on all five attempts for more tasks: 15, against nine. Astra’s scores also varied less between attempts on the same task.

More steps did not mean better results

Astra used a median of 33 model turns per scored run, compared with 160 for Opus and 166 for Gemini. Within each model, the relationship between turn count and score was weak.

However, a turn is not a standard unit of computation or cost. Providers had different tool interfaces, agent loops and output limits. The experiment therefore does not establish that fewer turns mean proportionally lower computational expense.

What the benchmark cannot settle

The final tasks were deliberately selected for difficulty against frontier models. Their scores should not be read as success rates for a representative developer workload.

The evaluation also excluded and reran attempts that hit step or output-token limits, alongside infrastructure failures. Such exclusions included 27 Opus runs and 55 Gemini runs. The reported scores consequently describe selected valid, completed attempts rather than counting every budget-exhausted attempt as a failure. Gemini lacked valid results on one task and was scored over 99.

Only 24 tasks are public; 76 remain held out to reduce exposure to future training data. That protection limits independent inspection. The underlying repositories are also largely public, so unpublished solutions do not guarantee unfamiliar source code.

The study is a preprint, not established evidence of workplace productivity. Its useful distinction is narrower: making substantial progress on a coding request and delivering a complete, acceptable change are different achievements.