What Do AI Coding Tokens Actually Measure?

Companies can count how many tokens an engineer’s AI tools consume. That makes token spend tempting as a measure of adoption, a way to set budgets, or even a productivity score. But tokens record activity, not what the activity accomplished. An agent may use many tokens exploring a codebase or revising an answer, whether or not that work ends in a commit.

A study by Shyam Agarwal and colleagues asks how closely token use tracks software output, and what happens to agent use that does not show up in repository records. The researchers linked five weeks of coding-agent telemetry to commit records for 404 engineers at JetBrains, a software development company. They also analyzed an anonymous company survey with 412 respondents. The survey could not be linked to individual engineers’ usage records.

The study compares engineers with themselves over time as well as with one another. That distinction matters: some people may habitually use agents more, while a particularly busy week may or may not bring a change in an individual’s usage. The researchers used statistical models to estimate these relationships, accounting for persistent differences between engineers and changes common to a given week. They also checked whether results changed when output was measured in other ways, such as lines changed or days with commits.

The short answer is that tokens and commits moved together only weakly. Within an engineer, a week with twice their usual commits was associated with about 20% more token use. In the reverse direction, a week with twice the usual token use was associated with roughly 10% more commits. These are associations, not proof that extra tokens produce extra commits—or that a token target would raise output.

Differences between people were striking even when commit counts were similar. Within each group of weeks with comparable commit counts, the 90th-percentile week’s token spend was 39 to 240 times the 10th-percentile week’s. Usage also tended to be a stable individual pattern: engineers who spent the most generally remained among the heavier users during the five weeks. Measures of repository activity explained 38% of differences in token use between engineers, but only 13% of week-to-week variation within an engineer. In other words, the records the researchers had captured only part of why token use varied.

That gap could reflect many things. Commits miss work such as design, code review, analysis, and experiments; they also do not tell us whether a change was useful or high quality. Conversely, some agent activity may be unproductive. The telemetry did not say what each agent session was about or connect sessions to particular commits, so the researchers could not directly sort productive non-commit work from wasted effort.

To estimate how much usage was not tied to that week’s commits, they used a model rather than a direct count. Their preferred estimates suggested that about 39% to 42% of tokens did not vary with weekly commit output. The paper treats this as an upper-bound-style estimate: it can include fixed costs of using an agent and agent use behind code that was ultimately written by a person. The more direct lower-bound measure—tokens in high-spend weeks with no commits—was much smaller, about 3%. These figures answer different questions and should not be read as a precise tally of wasted tokens.

Survey responses help explain why repository records are incomplete. In 89% of the coding-agent evaluations by technical respondents, people reported using agents for at least one task beyond coding, including data analysis, brainstorming, research, or planning. Such work may help someone understand a problem or explore an idea without producing a commit. Respondents also described costs: wrong or low-quality output, time spent steering agents, and verification work. These are self-reports, not independently measured gains or losses.

The authors’ practical conclusion is that token spend can help with resource accounting, but should not be treated as a measure of an engineer’s productivity or impact. To understand value, organizations would need to connect agent sessions to the tasks they support and assess outcomes beyond whether code was committed.

There are important limits. This is an observational study at one company, over five weeks, with voluntary tool use. JetBrains’ engineers and tooling may not represent other workplaces; commits are an incomplete measure of work; and the survey reached 15.8% of employees and relies on respondents’ recollections. Because engineers chose how much to use agents, the study cannot show what a spending mandate would do. The paper is an arXiv preprint, not evidence of peer review. Its results are a useful warning about a convenient metric, not a universal verdict on AI-assisted productivity.