Language models can exaggerate psychological patterns—and internal probes offer only part of the explanation

Predicting one questionnaire from another

Chih-Hao Hsu, Feng-Chun B. Chou and Pin-Hao A. Chen used questionnaire responses from 272 Taiwanese university students, aged 17–32. Participants completed seven psychological scales in Traditional Chinese, covering empathy, positive and negative emotions, well-being, personality, self-construal and loneliness. Together, the questionnaires contained 114 items grouped into 16 subscales.

The team tested 14 variants from the Llama 3 and Qwen 2.5 families, ranging from 0.5 billion to 14 billion parameters. These included both base models and instruction-tuned versions, which receive additional training to follow instructions.

For each participant, the researchers supplied answers from one questionnaire and asked the model to answer items from the other six as that person would. They repeated this with each questionnaire serving as the starting point. Answers were selected by comparing the model’s probabilities for the permitted numerical responses.

The main question was whether relationships across participants resembled those in the human data—not simply whether individual answers were correct.

Matching the pattern, overstating its strength

Larger models generally reproduced the pattern of human correlations more closely. But they also tended to exaggerate how strongly questionnaire scores were related.

The researchers separated pattern agreement from magnitude using a regression slope. A slope above one indicates amplified relationships relative to the human reference; it is not a measure of prediction accuracy. Across the nine models included in a later probe comparison, behavioral slopes ranged from 0.52 to 1.53.

Human questionnaires contain measurement noise, which can weaken observed correlations. Correcting for this in the seven-scale comparison reduced amplification but did not eliminate it: slopes remained between 1.19 and 1.39 for tested models with at least seven billion parameters.

That conclusion depends on the correction method. Applying the correction across all 16 subscales instead produced slopes below one. Several short personality subscales had low reliability estimates; restricting the analysis to more reliable subscales restored above-one slopes. The correction therefore does not provide a single, assumption-free verdict.

Looking inside the models

To examine internal organization, the team contrasted model activity under questionnaire profiles with high versus low responses on each subscale. These contrasts produced mathematical directions in activation space. The researchers then asked whether similarities among those directions resembled relationships among human questionnaire scores.

The strongest alignment appeared in larger instruction-tuned models. Qwen 14B Instruct reached a matrix-alignment correlation of 0.66, compared with 0.13 for its base counterpart. All seven base variants showed little alignment under this particular assay.

That does not establish that base models lack psychological information. A separate experiment trained statistical predictors, called ridge-regression probes, to recover questionnaire responses from internal activity immediately before an answer. Cross-validation tested predictions on held-out participants. These probes recovered cross-scale information even where the directional analysis found little organized structure.

Across 14 variants, stronger directional alignment was associated with greater behavioral amplification, with a correlation of 0.68. The association persisted after accounting for model size, but related variants from only two families do not constitute independent replications across the broader model landscape.

The measuring tool changes the picture

Probe-derived slopes ranged from 1.38 to 1.77, a narrower and higher range than the behavioral slopes. The authors call this measured probe–output gap “readout attenuation.” The term describes a difference between measurement pipelines, not an identified internal mechanism.

Crucially, synthetic-data experiments showed that ridge probing itself can inflate correlations, particularly when signals are noisy. An approximate adjustment left probe results associated with behavior, but its noise estimate came from a different analysis. It cannot precisely isolate amplification intrinsic to the model.

Exploratory interventions also changed internal directions to see whether answers shifted. Some produced psychologically structured effects, but results depended on the model, layer and direction-extraction method. They did not demonstrate that the measured directions mediate answers in the original prediction task.

What the evidence supports

This preprint supports a limited conclusion: psychological information is decodable, and some larger instruction-tuned models show internal geometry resembling this sample’s survey relationships. It does not show that models possess human psychological traits or validate individual psychological assessment.

The single student population, one language, post-hoc layer selection and measurement-sensitive results constrain generalization. Predicting population-level patterns—and sometimes exaggerating them—is not equivalent to reliably understanding a person.