Inside the Reasoning Model: What Its Hidden States Can—and Can’t—Tell Us

Looking beyond the final answer

A model can produce a correct answer without showing how it arrived at it. To investigate what may be happening inside, researchers can inspect the numerical signals passing through a model’s layers. This study asks whether those signals make it easier to distinguish correct from incorrect answers in models trained to reason with reinforcement learning (RL), compared with instruction-tuned models trained using supervised fine-tuning (SFT).

The distinction matters because a measurable internal difference is not automatically an explanation. If a simple statistical tool can read a signal from a model’s hidden states, that suggests the signal is present in a form the tool can detect. It does not, by itself, show that the signal causes better reasoning—or that the model uses it in the way a person might.

Three ways to look inside

The researchers studied four models from two families: DeepSeek-Math-7B-RL and DeepSeek-Math-7B-Instruct, and Olmo-3-Think and Olmo-3-Instruct. The DeepSeek pair offers the clearer RL-versus-SFT comparison. Both Olmo models underwent several kinds of post-training, including RLVR for Olmo-3-Think and SFT, DPO and RLVR for Olmo-3-Instruct. So the Olmo comparison cannot isolate the effect of RL alone.

First, the team created synthetic math word problems using five templates, including cost calculations and probability. For each model and problem, they recorded whether the answer was correct and examined the model’s hidden states at different layers. They trained simple linear classifiers—called probes—to predict correctness from those states. To avoid a probe succeeding simply because one answer type was much more common, they balanced correct and incorrect examples where possible. They also tested whether output length and truncation could predict correctness on their own.

Second, they used mean ablation on the two DeepSeek models: at each layer, they replaced the model’s activation with an average activation and measured the change in accuracy. Third, they examined how much output length varied across repeated answers to 300 math problems, with 50 responses per problem and model. They grouped problems by how often each model answered correctly.

Signals that do not settle the question

On some synthetic problem templates, probes predicted correctness more accurately for the RL-trained models than for their instruction-tuned counterparts. But the pattern was not consistent across comparisons. Crucially, output length and whether a response was cut short matched the DeepSeek RL probe’s performance on two templates. A probe may therefore be picking up features of response length or truncation, rather than a distinct, clear representation of correctness.

Probe accuracy also did not track the models’ own accuracy across the evaluable runs. That weakens the idea that better probe scores, by themselves, explain better math performance. The authors describe “clearer” correctness representations as a possible mechanism, not a demonstrated one.

The ablation results offer another clue, but a limited one. In DeepSeek-Math-RL, accuracy drops tended to increase with layer depth; the instruction-tuned model showed no similar trend. This could mean deeper layers matter more in the RL model. But the test used just 20 problems per model, making each measured change coarse. Ablating activations also does not directly establish what information those layers represent or how the model reasons.

Output-length variability showed different patterns across the two model families. In the DeepSeek pair, the RL model varied less than the instruction-tuned model on problems it answered correctly most of the time. In the Olmo pair, Think varied more than Instruct in that high-accuracy group. Across models, variability tended to be greater for problems with lower accuracy, though the relationship was not uniform: for DeepSeek-Math-Instruct, it depended on including truncated responses. These results do not support a simple rule that RL always makes response length more or less consistent.

A corrected, preliminary picture

The paper’s current version, arXiv v2, includes a correction note: an audit found errors in earlier probing and token-variability results, so those analyses were rerun. The ablation experiment was not rerun; its interpretation was revised. This makes it especially important to rely on the corrected account rather than earlier summaries of the work.

Other limits remain. The probes rely on synthetic problems, some labels and questions can be ambiguous, and models with very few wrong answers cannot be fairly compared using balanced data. The study also covers only a small set of model families and math tasks. Its results suggest useful avenues for testing how training shapes internal signals, but they do not establish that RL creates a particular reasoning mechanism or that such a mechanism causes improved performance. The paper is a preprint on arXiv; the supplied source does not establish peer-review status.