Making AI Forget—and Giving Providers Something to Audit

Machine unlearning aims to make a trained model behave as though selected training data had never been included. Retraining without that data provides an ideal reference, but the paper describes it as too expensive for practical use. The challenge is therefore not simply to suppress particular answers: it is to preserve useful capabilities while offering evidence that the requested change occurred.

In a preprint, Piyush Tiwary, Anushka Srivastava and Prathosh A.P. propose making that evidence an explicit engineering target. Their paper, titled “Widen the Gap, Hide the Gap: Auditable and Indistinguishable Unlearning for LLMs” in the supplied manuscript, adds auditability to three familiar evaluation concerns: forgetting, retention and resistance to membership inference. Their benchmark comparisons find that none of the eleven tested baseline methods satisfies all four dimensions simultaneously.

An internal signal, not a public giveaway

The authors define auditability through a comparison available to the provider, which holds both the original model and its modified counterpart. One audit statistic measures how much less likely the modified model considers the answer tokens in an example, relative to the original. Successful separation means examples requested for deletion receive different scores from examples retained in training.

They measure that separation using AUROC, a ranking statistic. A value of 1 means perfect separation on the evaluated examples; 0.5 means no separation. The evaluation selects the best of four audit statistics separately for each method, model scale and forget split. This is a specific operational definition of auditability, rather than a universal certification standard.

An outside attacker faces a different comparison. In the paper’s threat model, the attacker sees only generated text and tries to distinguish queries about removed data from a separate comparison set. Refusals or repetitive nonsense can become clues. The authors’ central proposal is to preserve a strong internal signal without directly exposing its behavioral signature.

Widen first, then screen the answers

The first stage, GapWiden, further trains an already modified model. A hinge loss pushes the average likelihood gap for forget-set answers beyond a chosen margin. A second loss anchors retained examples near the original model’s likelihoods. On TOFU, this stage is applied where baseline auditability falls below 0.95; the implementation applies it to all eleven methods on MUSE.

That clearer separation can worsen leakage when the modified model generates answers directly. GapVeil, the second stage, instead uses the original model to draft responses and the audit statistic to screen them. A draft above the threshold is rejected. Later drafts suppress content tokens from rejected attempts, while exempting prompt-shared tokens, special tokens and common function words and punctuation.

The implementation allows three resampling steps. If none passes, it returns the lowest-energy draft encountered. That fallback matters: the deployed procedure does not guarantee that every returned answer passes the audit threshold.

Stronger scores, uneven trade-offs

The experiments cover eleven unlearning methods, with TOFU evaluations at 1B, 3B and 8B model scales and MUSE evaluations using a 7B model. The appendix also reports RWKU results at 3B and 8B.

On TOFU’s forget10 split with the 8B model, average auditability rises from 0.944 to 0.997. Average forgetting improves from 0.904 to 0.927, while retention rises from 0.631 to 0.955. These forgetting and retention measures track whether answer content absent from the question reappears in generated responses; they should be understood in those terms.

MUSE exposes a sharper cost. On its News split, average auditability increases from 0.846 to 0.993 and forgetting from 0.575 to 0.887, but retention falls from 0.569 to 0.380. Books retention also declines, from 0.420 to 0.400. The authors attribute this to overlapping entities, wording and themes between forget and retain data, which can cause the screen to suppress valid retained content.

Privacy results also require qualification. Although the abstract describes oracle-level leakage performance, a separate behavioral analysis reports residual attacker advantages between 0.11 and 0.15 after GapVeil, where zero would indicate chance-level discrimination. The authors explicitly leave broader adaptive threat models for future verification.

Finally, the access restriction is essential. The original model remains part of generation, and both models’ weights are assumed confidential. The authors warn that exposing the engineered gap through model-weight access could enable exploitation of deleted data, making open-weight deployment particularly risky without other safeguards.

The contribution is therefore a concrete design for provider-side evidence and screened generation—not an unconditional proof of erasure or privacy. Its results show why auditability deserves separate measurement, while the retention losses and remaining attack signals show why that measurement cannot stand alone.