Tracing AI reasoning from final answers to internal activations

Starting with the outcome

In From Outcome to Representation: Tracing Reasoning Mechanisms through Integrated Policy Gradient, Changming Li and colleagues introduce Integrated Policy Gradient, or IPG. The reviewed version, arXiv:2602.02313v3, is hosted on arXiv and accepted at NeurIPS 2026. The method aims to connect a model’s final performance with internal activity across a generated solution, rather than identifying reasoning-related components solely because they activate near words such as “wait.”

The distinction matters because producing language associated with reasoning is not the same as solving a problem correctly. IPG instead uses an outcome signal, such as final-answer correctness, to rank internal components. The study reports both attribution results and experiments that alter the selected components. Its findings concern the tested models and settings, not an established general account of how language models reason.

How the method works

IPG borrows a technique from reinforcement learning called policy gradients. A language model generates each token according to probabilities shaped by the prompt and preceding text. Policy-gradient calculations connect the probability of those choices with feedback on the completed response, even when that feedback—such as a correct-or-incorrect judgment—is not directly differentiable.

Here, the researchers calculate gradients with respect to internal activations rather than updating model parameters. This produces estimates of how those activations relate to the outcome across the generation sequence.

The “integrated” part adds another step. Instead of measuring sensitivity only at the activation’s original value, IPG accumulates gradients along a path from a reference activation, usually zero, to the observed activation. The implementation approximates this integration with discrete steps. Scores are then aggregated across a support set of questions to select highly ranked components.

The analysis can target individual coordinates of internal representations or features extracted by a sparse autoencoder. This auxiliary model represents activations using a larger collection of sparsely active features, which can make patterns easier to inspect. Although IPG leaves the language model’s weights unchanged, the optional autoencoder does require separate training. Attribution also requires computation; “training-free” does not mean cost-free.

Once components are selected, the researchers multiply their activations by a scaling factor during generation. They also test removal by setting selected activations to zero.

What the experiments found

The main evaluations use Qwen2.5-Math-1.5B-Instruct and Llama3.1-8B-Instruct on mathematics and science-question benchmarks, including GSM8K, MATH500, AIME2024 and GPQA-Diamond. Most reported evaluations use greedy decoding, which selects the highest-probability next token. Additional experiments examine sampling and other models.

For Qwen, the sparse-autoencoder version increased reported MATH500 accuracy from 63.00% to 66.00%, and GPQA-Diamond accuracy from 24.75% to 35.35%. For Llama, MATH500 accuracy rose from 41.80% to 44.00%. These are results under the paper’s intervention settings, not guarantees for other tasks or deployments.

Removal experiments offer complementary evidence. Zeroing IPG-selected features reduced reported MATH500 accuracy from 63.00% to 28.80%, compared with 59.80% after random removal. This supports the selected components’ importance in that tested setting, without establishing that they exclusively implement reasoning.

The authors also report transfer between datasets and domains. Components identified on MATH500 improved performance when used for interventions on GSM8K and HumanEval coding problems. A separate experiment modified output length in a distilled DeepSeek model while maintaining similar accuracy across several tested scaling factors. Output length, however, is a behavioral measurement, not a direct measurement of reasoning quality.

An incomplete map of internal computation

The analyses suggest that influential components are concentrated in particular layers rather than distributed uniformly. Some selected autoencoder features activate in contexts involving planning, arithmetic or summarization. Their activity also varies across the generated response.

Those descriptions need restraint. Semantic labels come from inspecting highly activating text contexts. Temporal analysis uses paragraph breaks and normalized generation progress, rather than independently verified boundaries between reasoning operations. These observations do not establish a complete sequence of specialized internal reasoning stages.

IPG also approximates causal mediation rather than measuring it exactly. Its selected neurons overlap more strongly than comparison methods with a reference constructed through group-based zero-ablation experiments. That reference is itself a proxy, not ground truth about every component’s role.

The authors acknowledge that coarse correctness signals and reward-model judgments can introduce noise or bias. Comparisons with conventional attribution methods also use different attribution objectives, limiting what performance differences reveal about the integration technique alone.

The contribution is therefore a practical way to nominate and test outcome-relevant internal components. The reported interventions strengthen the evidence beyond activation correlations, but they do not demonstrate a universal reasoning mechanism or make a model’s written explanation a faithful record of its internal computation.