Could the shape of word connections help evaluate AI reasoning?

In the preprint “Reasoning is topologically simple,” Kapulkin and Kershaw investigate whether the way words connect across sentences can reveal patterns associated with reasoning. Their four experiments find statistical associations, while leaving open whether the measure captures reasoning quality itself or other features of writing.

An AI’s written reasoning can be long and difficult to assess. One approach is to ask another language model to inspect it, but that introduces another model’s costs, biases and safety concerns. A new preprint explores a different possibility: measuring the mathematical shape of connections among words.

The authors’ hypothesis is that reasoning tends to combine concepts rather than merely introduce them. In their representation, this could produce fewer persistent loops of connections left unfilled. Their experiments identify patterns consistent with that explanation, but do not establish a general test of reasoning quality or correctness.

Turning sentences into shapes

The method reads a text sentence by sentence, after removing a fixed list of common words. Each remaining word becomes a point. Words appearing together in a sentence are connected by edges; three appearing together also create a filled triangle. Larger groups create higher-dimensional counterparts.

Imagine three words that have appeared together in pairs, but never all in one sentence. Their connections can form a triangular loop with an empty interior. A later sentence containing all three fills that triangle. More generally, the researchers track one-dimensional holes: loops not filled by the triangles accumulated so far.

These are holes in a mathematical representation of word use, not demonstrated gaps in an argument. The proposed connection to reasoning is an interpretation that requires further testing.

The technique, called co-occurrence persistent homology, records when holes appear and disappear. Its total persistence is the sum of their lifetimes, measured in sentence steps. Holes remaining at the end are assigned an endpoint one step beyond the final sentence.

To account for vocabulary and text length, the authors divide this total by the mean from 200 randomized versions. Those versions shuffle sentence lengths and redistribute words into sentences. A score below one means simpler topology than this randomized baseline; lower scores indicate greater simplicity under the measure. Unlike approaches based on numerical embeddings, this construction directly uses sentence order and word co-occurrence.

Four comparisons

First, the researchers examined 24 Project Gutenberg texts in each of four categories, truncating every text to 300 sentences. Mathematics and argumentative philosophy represented reasoning-focused prose; history and short stories represented prose not primarily organized as structured reasoning.

Median normalized scores were 0.537 for mathematics, 0.674 for philosophy, 0.730 for history and 0.872 for short stories. All four planned comparisons between reasoning-focused and other categories were statistically significant. This supports a difference between the sampled genres, rather than proving that every mathematical passage reasons better than every narrative.

Second, the study compared 135 AI-written mathematics papers released by OpenAI with human-written papers in the same fields, published in leading journals. Claude Opus 5.5 performed the matching. Each pair was truncated to the shorter paper’s sentence count.

The human paper had simpler topology in 126 of 135 pairs, approximately 93%. Median scores were 0.583 for human papers and 0.722 for AI papers. The authors suggest that more refined presentations of reasoning might explain the difference, while explicitly acknowledging alternatives such as writing style.

Third, the researchers analyzed reasoning traces from 48 open-weight models on 30 AIME mathematics problems. They retained the first 100 reasoning sentences and discarded shorter traces, leaving some models with fewer usable examples.

Across models, higher Epoch Capabilities Index scores were moderately associated with lower median topological complexity. The reported Spearman correlation was −0.444, with a one-sided p-value of 0.0008. This was a model-level association with a capability index, not a demonstration that the statistic identifies correct individual answers.

Finally, the authors tested deepseek-v4-flash-0731 in 26 SHADE-arena environments. Each environment had a benign condition and a condition prompting a covert sabotage side task alongside the main task. They ran three trials per condition and matched sentence counts within each environment.

Benign reasoning had simpler average topology in 19 of 26 environments. Mean normalized scores were 0.363 for benign runs and 0.449 for sabotage runs, with a one-sided Wilcoxon p-value of 0.009.

A possible signal, not a verdict

The safety result does not establish a working sabotage detector. Individual-run discrimination and false-positive rates remain unevaluated, and that experiment used only one model.

More broadly, genre conventions, vocabulary, editing and model-specific styles could contribute to the findings. The method treats different word forms separately, loses word order within sentences and has not been tested for robustness to alternative common-word lists.

The preprint therefore motivates further investigation of a structural signal. Direct assessments of reasoning quality and readability, with appropriate controls, are still needed before topological simplicity can be treated as evidence of better reasoning rather than a correlated feature of its presentation.