An AI’s Financial Explanations May Not Match Its Internal Signals

Asking an AI to become more cautious sounds like a straightforward way to simulate a risk-averse investor. A financial research preprint suggests that such instructions can also change unrelated internal concepts—and sometimes disrupt otherwise correct decisions.

A language model can explain a financial decision fluently without faithfully describing the internal signals behind it. That distinction is central to A Financial Brain Scan of the LLM, an arXiv preprint by Hui Chen, Antoine Didisheim, Mo Pourmohammadi, Luciano Somoza and Hanqing Tian.

The researchers examine whether tools called sparse autoencoders can make a model’s financial processing more interpretable and controllable. Their main experiments use Gemma-2-9B-IT with pretrained Gemma Scope autoencoders. The findings concern these experimental settings, rather than establishing a universal account of how AI reasons.

Turning internal activity into economic labels

Language models represent information through dense internal vectors, with concepts distributed across overlapping patterns. A sparse autoencoder expands that representation into a larger collection of features, only a small subset of which activates for a given input. Researchers can inspect those features or alter the model’s internal state along their associated directions without retraining the underlying language model.

The authors start with 131,072 features. They define 46 economic topics, generate synthetic examples for each using another language model, and select features that respond strongly and selectively. This produces 7,278 features grouped into five broad themes, including corporate finance and macroeconomics.

To check the labels, they hold a neutral prompt constant while steering topic-associated features. The resulting vocabulary generally matches the intended topic: derivatives steering, for example, elicits terms associated with options.

The “brain scan” remains an analogy. Its visual map uses economic labels to help organize features, so the apparent clusters are not independent proof that the model naturally divides into those economic regions.

Asking for caution changes more than caution

The paper compares two ways to alter simulated risk preferences. Prompt steering adds instructions such as becoming more risk-averse. Autoencoder-based steering instead adds a numerical perturbation associated with a selected risk-aversion feature.

In choices between a risky investment and a safer alternative, both methods shift preferences toward safety. But prompt instructions produce abrupt probability changes and substantial changes in other measured economic topics. Direct feature steering produces smoother adjustments in the tested settings.

A second experiment makes the distinction more consequential. The model chooses between projects with identical risk but net present values of $12 million and $7 million. Because risk is equal, greater risk aversion should not reverse the preference for the higher-value project. Stronger prompt steering nevertheless causes such a reversal. Direct feature steering preserves the correct choice across the tested intervention strengths.

These results challenge the assumption that assigning a preference through words changes only that preference. They do not show that direct steering is perfectly isolated or reliable at arbitrary strengths: the appendix notes that extreme steering can produce incoherent text.

Explanations agree unevenly with internal attribution

For macroeconomic forecasts, the authors combine feature activation with gradients measuring how sensitive the model’s answer probabilities are to those features. These are local, first-order attribution estimates—not a complete reconstruction of the model’s reasoning.

They compare those estimates with the model’s own topic-importance reports. Agreement varies considerably. Across 46 topics, the correlation reaches 0.968 for consumer-price inflation but only 0.105 for AAA bond rates. Four forecast targets have correlations that are not statistically significant.

The evidence therefore supports a qualified warning: self-reports can align closely with internal attribution for some tasks, yet provide a weak guide for others.

Historical financial results need careful interpretation

The authors also study 2,108,120 overnight Reuters alerts about U.S.-listed firms from 2015 through 2024. Suppressing a positive-sentiment feature reduces the share classified as positive from 67.6% to 58.4%. A corresponding long-short strategy’s annualized Sharpe ratio, a risk-adjusted performance measure, rises from 4.24 to 4.51. Negative prompt steering instead drives positive classifications close to zero and substantially weakens portfolio performance.

In a separate predictive-model comparison, all 7,278 labeled features produce an equal-weighted portfolio Sharpe ratio of 5.22, versus 3.67 using dense embeddings from the same model. Value-weighted results are lower: 1.99 versus 1.55.

These historical comparisons do not settle training-data contamination concerns. The steering comparison assumes interventions do not change reliance on memorized information. For reduced feature sets, rankings are also calculated across all annual training windows, rather than selected independently using only information available at each historical date.

The preprint’s strongest message is methodological: researchers should measure what their interventions change, rather than assume a verbal persona isolates an economic trait. Its financial results illustrate the potential of that approach without establishing dependable future investment performance.