Can an AI Model Forget—or Just Stop Answering?

A model trained to stop discussing a subject may give no useful answers while still retaining information that an adversarial technique can recover. That distinction matters for machine unlearning: attempts to remove selected knowledge without retraining an entire language model. Evaluating only the answers risks confusing successful suppression with actual deletion.

In the arXiv preprint The Geometry of Ignorance, Adar Peleg, Dvir Alsheich, Tomer Ashuach and Avi Mendelson propose examining hidden activations instead. Their framework, called activation fragmentation, measures whether representations associated with a subject point in consistent directions or scatter across activation space. The results support a potential diagnostic tool, not a definitive test of whether a model has forgotten.

Looking for a concept’s internal footprint

The method compares an input about a target concept with an anonymized version. Names and other identifying terms are replaced: Harry Potter might become Alex Ryden, for example. Subtracting the anonymized input’s hidden-state vector from the original produces what the researchers call an anon-difference vector.

The intended effect is to reduce shared linguistic structure while retaining a signal associated with conceptual identity. Across many examples, the authors hypothesize that familiar concepts produce more coherent directions, whereas unfamiliar ones produce fragmented geometry.

The implementation details matter. For the knowledge-detection experiment, inputs contain both questions and supplied answers, and substitutions apply to both. The measurement therefore examines representations of provided material, rather than simply watching a model internally prepare an answer. With one anonymized counterpart, extracting the vectors requires two forward passes per input and no generated tokens.

The researchers measure fragmentation through several statistics. Entropy describes how broadly representations occupy different directions. Their Activation Uniformity Scatter Scores compare normalized average directions from separate batches of examples. These are related but distinct geometric measurements—not interchangeable readings of a single quantity called knowledge.

Separation is clearer than universal recovery prediction

The first experiment compares 96 Harry Potter question–answer pairs with 100 pairs about fictitious authors from the TOFU benchmark across ten instruction-tuned models from the Qwen, Gemma and LLaMA families. The authors report strong separation, particularly for their batch-direction dispersion metrics. Additional comparisons involving Star Wars and another Harry Potter split extend the evidence beyond the original contrast.

However, the headline results use layers selected to maximize known-versus-unknown separation. That selection makes them different from measurements taken at a predetermined layer on an unseen task. The paper also reports encouraging results from a fixed relative-depth heuristic in a separate nine-model analysis. Neither result establishes that these benchmarks capture every form or degree of knowledge.

The second experiment studies recovery after five unlearning methods, using three models and three target concepts: Harry Potter, Star Wars and William Shakespeare. Four attack types probe the resulting models: activation steering, in-context examples, gradient-optimized suffixes and membership inference.

Across 44–45 post-unlearning runs with complete attack data, two batch-direction dispersion metrics correlate negatively with in-context recovery, with Spearman correlations around −0.50. More fragmentation accompanies less recovery through that route. Other measurements behave differently: raw-activation entropy correlates positively with gradient-suffix attack success, while anon-difference entropy correlates positively with steering success. No metric survives the reported multiple-testing correction for membership inference.

Restricting the analysis to runs meeting the paper’s behavioral unlearning criteria further narrows the result. In-context recovery is the only attack for which the examined metric family shows both leave-one-run-out sign stability and statistically significant relationships after correction. There is no universal rule that greater fragmentation means stronger protection against every attack.

An audit signal, not an erasure certificate

The paper acknowledges that there is no direct oracle for what a model knows. Fictitious benchmark entities serve as an ignorance proxy; they do not provide the certainty of controlled pretraining that demonstrably excluded the material.

Anonymization can also introduce lexical and tokenization artifacts. The mathematical connection between low entropy and same-direction clustering requires an additional non-negative-cosine assumption: low entropy alone can describe vectors pointing in opposite directions.

Reporting inconsistencies add uncertainty. The main text and appendix differ on the unlearning experiment’s layer-selection procedure and use of multiple anonymized variants. Some comparisons with prior methods also pair geometry from retrained checkpoints with attack scores from the original checkpoints.

The practical proposition is therefore limited but useful: hidden-state geometry may complement output tests and help prioritize adversarial evaluation. It requires access to internal activations, not merely a conventional text-only interface. As presented in this preprint, it offers evidence about recoverability under tested conditions—not proof of permanent deletion.