A Financial Brain Scan for AI—and a Warning About Its Explanations

A model can sound like a cautious investor without changing only its appetite for risk. A new preprint looks inside that discrepancy—and asks what researchers can learn by intervening on the model’s internal features instead.

Ask a language model to behave like a risk-averse investor, and it may oblige. But what, exactly, has changed? Is the model expressing a single preference, or has the instruction also altered its attention to a host of other financial ideas?

In the preprint A Financial Brain Scan of the LLM, Hui Chen and coauthors explore that question with an interpretability tool called a sparse autoencoder, or SAE. Their central argument is that examining—and sometimes manipulating—internal features can reveal complications hidden by a model’s fluent answers. The work is promising, but its “brain scan” is an analogy, not a literal reading of machine thought.

A map made for finance

An SAE is an auxiliary network that translates a language model’s dense internal state into a larger representation in which relatively few features are active at once. The authors use a pretrained SAE with Gemma-2-9B-IT, leaving the language model’s weights unchanged.

To make the feature dictionary useful for finance, they start with 46 economic topics, generate example phrases for each, and identify features that respond strongly and selectively. Their procedure selects 7,278 features from a dictionary of 131,072. When they steer selected features while holding a neutral prompt fixed, the model often produces vocabulary associated with the assigned topic: derivatives steering, for example, elicits terms such as “option” and “call.”

That is useful validation, not proof that each feature is a perfectly isolated economic concept. Labels are inferred from responses to synthetic examples, and the authors’ forecast contributions combine feature activations with gradients—a local, first-order measure of influence. The map is an analytical instrument, not a complete inventory of everything the model represents.

When a persona changes more than a preference

The paper’s sharpest lesson concerns simulated economic agents. Researchers may assign a model a trait by writing it into a prompt, then interpret differences in its decisions as the effect of that trait. The authors test that assumption by asking the model to choose between riskier and safer investments.

Both a risk-aversion instruction and an intervention on an SAE feature shift the model toward the safer option. But the prompt also changes other measured topic activations, including beta and passive investing. In the tested choice, the largest collateral topic change is about half the change associated with risk aversion. The model’s choice probabilities also move abruptly as the wording’s stated percentage changes, whereas direct feature steering produces a more gradual response.

The distinction matters beyond appearances. In another test, two projects have identical risk but different net present values. Risk aversion should not make the lower-value project preferable. At strong prompt-steering settings, the model nevertheless selects it; across the tested SAE-steering range, it continues to choose the higher-value project. Changes in other topic activations accompany the prompt-induced reversal, although that coincidence alone does not establish which collateral change caused it.

Direct intervention therefore offers a more controlled experiment in these settings—not a guarantee that a feature captures risk aversion alone or that results will transfer to every model and task.

Explanations are evidence, not an audit trail

The authors also compare internal attributions with the topics the model says it used when making macroeconomic assessments. Agreement varies considerably. Across the 46 topics, the reported correlation is very high for consumer-price forecasts but weak and statistically insignificant for several targets, including the three-month Treasury bill rate and AAA bond rate.

This is a more nuanced result than a blanket verdict that all model explanations are useless. Some align closely with the scan; others do not. Nor is the scan itself a perfect ground truth. Still, the comparison is a reason to treat a polished explanation as a claim to investigate, rather than a faithful transcript of the model’s internal process.

A promising result with financial caveats

The authors also test interventions on forecasting and news classification. Increasing a feature associated with quantitative easing changes a crisis-era Treasury bill forecast toward a decline, closer to professional forecasters’ revisions. Reducing a positive-sentiment feature makes the model classify fewer firm-news alerts as positive; in the reported long-short strategy, the annualized Sharpe ratio rises from 4.24 to 4.51.

A separate return-prediction exercise reports an equal-weighted Sharpe ratio of 5.22 using all labeled SAE features, compared with 3.67 using dense final-layer embeddings. These are results for the paper’s particular sample and design, not assurances of future trading performance. The paper notes potential training-data contamination; its comparison of steered strategies relies on the assumption that steering does not change reliance on memorized information. Moreover, reduced-feature rankings use coefficients averaged across all training windows, including later ones, a qualification when interpreting those reduced-feature results as strictly out of sample.

The editorial takeaway is not that finance has acquired a transparent artificial analyst. It is that better instruments can make familiar questions harder to ignore: what did an instruction actually change, what does an explanation really tell us, and how much confidence should an attractive backtest command? This preprint supplies inventive ways to investigate those questions, while leaving their broader answers open.