Lowering an AI’s Odds of Revealing Personal Data May Not Prove Targeted Protection

A lower probability of producing personal information does not necessarily demonstrate that a language model’s privacy-related mechanisms have been selectively suppressed. That is the central argument of a November 3, 2026 arXiv preprint by Chaerin Lee, Wonho Lee, Sohee Park and Daeseon Choi of Soongsil University. The study is a preprint, not an established peer-reviewed finding.

Across 36 experimental configurations, the researchers report that 17 appeared successful when judged solely by a reduction in the log-probability of the correct personal-information tokens. Only one passed their stricter evaluation, which also considered token rankings and comparisons with matched random interventions.

Targeting a feature—or disturbing the model?

The study examines sparse autoencoders, or SAEs: tools that decompose a model’s internal activations into sparsely active features. Researchers can identify features associated with personally identifiable information, or PII, and modify the model’s internal state along their corresponding directions.

The concern is that a sufficiently large internal disturbance might reduce the probability of a sensitive answer even without targeting a privacy-specific feature. A probability decrease alone cannot separate those explanations.

The team tested Llama-3.1-8B-Instruct, Qwen3-8B and Gemma-2-9B, all fine-tuned using LoRA on a synthetic medical PII corpus. The four information types were names, patient IDs, phone numbers and email addresses. No real patient records were used.

For each model, the researchers trained SAEs at three depths. Combining three models, three layers and four information types produced the 36 configurations. Features were selected on a discovery split according to which candidate most reduced the correct PII span’s log-probability. Final evaluation used separate pairs, with 50 pairs tested across nine prompt templates per configuration, including Korean labels, JSON and narrative formats.

Why the random controls matter

The main control preserved the selected intervention’s token-by-token magnitude but replaced its direction with a random one. It also maintained a shared direction across the intervened positions within each forward pass. This comparison asked whether the chosen feature’s direction mattered beyond the size and pattern of the disturbance.

The researchers measured both changes in the correct PII span’s log-probability and changes in a rank-based proxy for extraction difficulty. The latter tracks whether the correct tokens move further down the model’s candidate rankings; it is not a direct extraction attack.

Their strict criterion required a log-probability decrease of at least 0.3, a rank-cost increase of at least one, and effects at least five times the random control’s magnitude on both measures. Only name suppression at Llama’s layer 16 passed. Relaxing the selectivity threshold to two admitted three configurations, still far fewer than the 17 identified by log-probability alone.

Two problems emerged. First, random directions sometimes suppressed the target more strongly than the selected feature. For Gemma’s layer-20 name configuration, all 20 additional control directions sampled from the same SAE dictionary produced stronger suppression. That check addressed whether the result depended solely on using random directions unrelated to the learned dictionary.

Second, at three model-layer combinations, the selection procedure chose the same feature for all four PII types, despite markedly different intervention effects across those types. Shared selection therefore did not establish a shared privacy-specific mechanism.

A promising exception, not a privacy guarantee

The Llama layer-16 name configuration stood apart: none of its 20 sampled dictionary-direction controls produced comparable suppression. In a limited utility check covering three configurations and three benchmarks, its accuracy changed by no more than one percentage point. The other two tested configurations lost roughly 12–34 percentage points, illustrating how apparent suppression can accompany broader performance damage.

A separate feasibility study used activation-rate-selected candidates rather than probability-based selection. Across 9,000 feature-and-pair intervention runs, the researchers compared zeroing individual features with matched random disturbances. Among 150 candidates meeting their total-effect magnitude threshold, 146 had residual-effect ratios within the study’s specified range for being indistinguishable from random. The authors interpret this as support for seeking better causal candidates through circuit-level analysis.

The conclusions remain bounded. The experiments covered synthetic medical data, three 8B/9B-class models, three layers per model and single-feature interventions. Other SAE capacities, selection methods and multi-feature approaches remain untested. Full adversarial extraction attacks were not evaluated, and utility testing was limited.

The practical lesson is methodological: reduced output probability is evidence of an effect, but not sufficient evidence of targeted privacy protection. The authors propose combining stronger mechanism-specific diagnostics with end-to-end extraction testing, rather than treating either as a complete privacy guarantee.