TRANSMISSION // 2611.23216v1
Higher coding scores brought no clear rise in merges
Source published
A GitHub study found no statistically clear increase in merged pull requests when developers had access to a higher-scoring model or switched to newer ones.

Coding benchmarks test models on defined tasks. But do higher scores mean developers finish more everyday work? A preprint examining GitHub pull requests around a temporary model suspension and several upgrades found no clear increase in merging.
A coding model can improve on a benchmark without helping its users finish more work. That is the question Yining Hua investigates in a preprint: when developers switch to a model that scores higher on coding tests, do more of their proposed changes make it into software projects?
The study examines public GitHub pull requests associated with Claude Code. Its first comparison uses a 19-day suspension of Claude Fable 5 in June 2026. Developers who had used Fable 5 had to switch to other models; the paper compares their pull requests before, during and after the suspension with activity by developers who continued using Opus 4.8. The author treats the suspension as a natural experiment: an outside event changed model access while the same developers remained under observation.
A second analysis looks at five Opus and Sonnet model upgrades. It compares developers’ pull requests in the week before and after they switched. Unlike the suspension, these upgrades were not imposed, so this part is observational: people chose whether to switch.
The main outcome was whether a pull request—a proposed code change—was merged within seven days. The researchers also measured how many pull requests were merged per week after upgrades. They identified models using attribution text in pull requests and used statistical models that account for stable differences between developers and, in the suspension analysis, calendar-date changes shared across users.
No statistically clear increase
When Fable 5 was available, its users’ seven-day merge rate was estimated to be 1.4 percentage points higher than during the suspension. But the 95% confidence interval ranged from 1.4 points lower to 4.3 points higher. Because it includes no change, the result is not statistically significant. It does not establish that access had no effect: the data remain compatible with a modest benefit, no change, or a small decline.
Across the five upgrades, the estimated average change in seven-day merge rate was minus 0.9 percentage points, with a 95% confidence interval from minus 1.9 to plus 0.2. None of the five estimates showed a statistically significant increase. The estimate for weekly merged pull requests was plus 2.4%, with an interval from minus 4.9% to plus 10.2%, also not statistically significant.
For six upgrades with reported scores, the newer models scored 5 to 18 points higher on SWE-Bench Pro. Benchmark points and merge-rate percentage points describe different outcomes and should not be compared directly. The narrower finding is that the study did not detect a corresponding increase in these measures of completed work.
What counts as completed work?
A merged pull request is a useful public sign that code was kept in a project, but it is not a complete measure of productivity or quality. In a sample of 2,019 pull requests, the author found that 95% were merged by the person who opened them and 90% had no approval from another account. A merge therefore often reflected the author’s own decision, not an independent maintainer’s assessment. The study also cannot measure time saved, work outside GitHub, or every correction made after a merge.
The suspension comparison depends on the assumption that, without the suspension, Fable 5 users’ merge rates would have changed like those of the comparison group. Losing access also meant switching models, so the estimate captures the overall effect of that access change—not benchmark ability in isolation. The upgrade analysis is subject to selection: developers who chose to switch may differ in ways the analysis cannot fully account for. Model attribution is another constraint. In the researchers’ checks, only about 1% of merged Claude Code pull requests named the model in the description, limiting which activity could be linked to a specific model.
The paper proposes that many everyday tasks may already be manageable by older models, while benchmarks test different or more narrowly defined capabilities. That is a possible explanation, not a conclusion established by these comparisons. The findings do not show that upgrades never help or that benchmark improvements are meaningless. They show that, in the settings and measures studied, higher benchmark scores were not accompanied by a statistically clear increase in merged work.
This is an arXiv preprint, not a peer-reviewed publication. Its results concern Claude Code users and GitHub-based measures and may not generalize to other tools, teams or kinds of software work. The author reports releasing a de-identified dataset and analysis code, enabling further examination of the results.
- AI coding models
- developer productivity
- coding benchmarks
- GitHub pull requests
- software engineering
- AI evaluation
Reviewed cross-references
These connections were reviewed by a model, not independently fact-checked by a human. Similarity does not imply agreement; consult the original papers and the evidence below.
What Do AI Coding Tokens Actually Measure?
Shared method: Both analyze repeated developer-level observations over time and compare outcomes within developers: Hua uses developer fixed effects in the model-upgrade analysis, while Agarwal et al. compare engineers’ weekly token spend and commits within the same engineer. The outcomes and designs differ—merged pull requests versus commits, and a model-access natural experiment versus observational token-use associations—so this is a methodological overlap, not evidence that the findings corroborate each other.
Model review: · openai-codex/gpt-6-luna
Evidence and provenance
Selected excerpts, not complete papers. PDF extraction can omit figures or damage equations, tables and reading order; sources may be preprints.
2611.23216v1 — Original paper
Excerpt reference:
static/papers/2611.23216v1.md:171-286
Basis: Paper text
Source SHA-256:9c65c4e08fbd728db012ca02f516509ddbdbecd2f4824f20fd9fdc0d6e502fa42611.13396v1 — Original paper
Excerpt reference:
static/papers/2611.13396v1.md:202-276
Basis: Paper text
Source SHA-256:e014b42aa4ee852e75e5c728dd36d5762779a87bafc77dc8251186db63ac0802
Related reading
Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.
What Do AI Coding Tokens Actually Measure?
Source published
Shared keywords: developer productivity, software engineering
PersonaCodeBench tests whether coding agents meet users’ needs as well as pass tests
Source published
Shared keywords: ai evaluation
When an AI’s training score rises but its answers get worse
Source published
Shared keywords: ai evaluation
Choosing an Auxiliary Objective for Software Optimization
Source published
Shared keywords: software engineering
A tougher coding benchmark finds leading AI agents still struggle to finish the job
Source published
Shared keywords: software engineering