TRANSMISSION // 2611.23216v1

Higher coding scores brought no clear rise in merges

Source published

A GitHub study found no statistically clear increase in merged pull requests when developers had access to a higher-scoring model or switched to newer ones.

AI-generated conceptual illustration for Higher coding scores brought no clear rise in merges
AI-generated conceptual illustration; not documentary evidence.

Coding benchmarks test models on defined tasks. But do higher scores mean developers finish more everyday work? A preprint examining GitHub pull requests around a temporary model suspension and several upgrades found no clear increase in merging.

A coding model can improve on a benchmark without helping its users finish more work. That is the question Yining Hua investigates in a preprint: when developers switch to a model that scores higher on coding tests, do more of their proposed changes make it into software projects?

The study examines public GitHub pull requests associated with Claude Code. Its first comparison uses a 19-day suspension of Claude Fable 5 in June 2026. Developers who had used Fable 5 had to switch to other models; the paper compares their pull requests before, during and after the suspension with activity by developers who continued using Opus 4.8. The author treats the suspension as a natural experiment: an outside event changed model access while the same developers remained under observation.

A second analysis looks at five Opus and Sonnet model upgrades. It compares developers’ pull requests in the week before and after they switched. Unlike the suspension, these upgrades were not imposed, so this part is observational: people chose whether to switch.

The main outcome was whether a pull request—a proposed code change—was merged within seven days. The researchers also measured how many pull requests were merged per week after upgrades. They identified models using attribution text in pull requests and used statistical models that account for stable differences between developers and, in the suspension analysis, calendar-date changes shared across users.

No statistically clear increase

When Fable 5 was available, its users’ seven-day merge rate was estimated to be 1.4 percentage points higher than during the suspension. But the 95% confidence interval ranged from 1.4 points lower to 4.3 points higher. Because it includes no change, the result is not statistically significant. It does not establish that access had no effect: the data remain compatible with a modest benefit, no change, or a small decline.

Across the five upgrades, the estimated average change in seven-day merge rate was minus 0.9 percentage points, with a 95% confidence interval from minus 1.9 to plus 0.2. None of the five estimates showed a statistically significant increase. The estimate for weekly merged pull requests was plus 2.4%, with an interval from minus 4.9% to plus 10.2%, also not statistically significant.

For six upgrades with reported scores, the newer models scored 5 to 18 points higher on SWE-Bench Pro. Benchmark points and merge-rate percentage points describe different outcomes and should not be compared directly. The narrower finding is that the study did not detect a corresponding increase in these measures of completed work.

What counts as completed work?

A merged pull request is a useful public sign that code was kept in a project, but it is not a complete measure of productivity or quality. In a sample of 2,019 pull requests, the author found that 95% were merged by the person who opened them and 90% had no approval from another account. A merge therefore often reflected the author’s own decision, not an independent maintainer’s assessment. The study also cannot measure time saved, work outside GitHub, or every correction made after a merge.

The suspension comparison depends on the assumption that, without the suspension, Fable 5 users’ merge rates would have changed like those of the comparison group. Losing access also meant switching models, so the estimate captures the overall effect of that access change—not benchmark ability in isolation. The upgrade analysis is subject to selection: developers who chose to switch may differ in ways the analysis cannot fully account for. Model attribution is another constraint. In the researchers’ checks, only about 1% of merged Claude Code pull requests named the model in the description, limiting which activity could be linked to a specific model.

The paper proposes that many everyday tasks may already be manageable by older models, while benchmarks test different or more narrowly defined capabilities. That is a possible explanation, not a conclusion established by these comparisons. The findings do not show that upgrades never help or that benchmark improvements are meaningless. They show that, in the settings and measures studied, higher benchmark scores were not accompanied by a statistically clear increase in merged work.

This is an arXiv preprint, not a peer-reviewed publication. Its results concern Claude Code users and GitHub-based measures and may not generalize to other tools, teams or kinds of software work. The author reports releasing a de-identified dataset and analysis code, enabling further examination of the results.

Reviewed cross-references

These connections were reviewed by a model, not independently fact-checked by a human. Similarity does not imply agreement; consult the original papers and the evidence below.

  • What Do AI Coding Tokens Actually Measure?

    Shared method: Both analyze repeated developer-level observations over time and compare outcomes within developers: Hua uses developer fixed effects in the model-upgrade analysis, while Agarwal et al. compare engineers’ weekly token spend and commits within the same engineer. The outcomes and designs differ—merged pull requests versus commits, and a model-access natural experiment versus observational token-use associations—so this is a methodological overlap, not evidence that the findings corroborate each other.

    Model review: · openai-codex/gpt-6-luna

    Evidence and provenance

    Selected excerpts, not complete papers. PDF extraction can omit figures or damage equations, tables and reading order; sources may be preprints.

    • 2611.23216v1 — Original paper

      Excerpt reference: static/papers/2611.23216v1.md:171-286
      Basis: Paper text
      Source SHA-256: 9c65c4e08fbd728db012ca02f516509ddbdbecd2f4824f20fd9fdc0d6e502fa4

    • 2611.13396v1 — Original paper

      Excerpt reference: static/papers/2611.13396v1.md:202-276
      Basis: Paper text
      Source SHA-256: e014b42aa4ee852e75e5c728dd36d5762779a87bafc77dc8251186db63ac0802

Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.