When Compressing AI Context Weakens Refusals

The safety question hidden in compression

Context compression gives a language model a compact representation of input rather than all the original text. It can reduce inference costs and help systems work with longer material. One question is whether a compressed representation can preserve not just the information in an input, but also the model’s expected behavior when responding to it.

Nguyen and Lin’s preprint tests a narrow but consequential version of that question: does a model still refuse harmful requests when it sees only a compressed representation of them? Across the models and compression methods tested, refusal rates fell, though by differing amounts.

The distinction between this experiment and a typical deployment matters. Many document-grounded systems compress external material while leaving the user’s question in ordinary text. Here, the researchers compressed the entire user request and withheld the original prompt from the decoder. This was a deliberate stress test of a compression-first pipeline, not evidence that every system using compressed documents has the same vulnerability.

Refusal rates fell, by different amounts

The researchers compared uncompressed prompts with two encoder-based approaches: fixed-length In-Context Autoencoders (ICAEs) and the variable-length ARC-Encoder. Their custom dataset contained 624 model-generated prompts, half benign and half harmful. Half the dataset was reserved for evaluation; the other half was used to develop the intervention. An automated model judged whether responses were refusals.

On the held-out set, Mistral-7B-Instruct-v0.2 refused 62.82% of harmful prompts without compression and 10.9% with ICAE compression. LLaMA-2-7B-Chat fell from 97.44% to 85.9% under ICAE. With ARC, LLaMA-3.1-8B-Instruct fell from 96.15% to 36.54%, while Mistral-7B-Instruct-v0.3 fell from 20.51% to 8.97%. Refusal rates on benign prompts remained at or near zero in these comparisons.

The pattern also appeared on HarmBench and AdvBench, two established safety benchmarks, although its size varied. On HarmBench, for example, ICAE-compressed Mistral-v0.2 fell from 57.5% to 16.5%, while ARC-compressed LLaMA-3.1 fell from 84% to 47.5%. The ARC encoders were additionally fine-tuned for use with the instruction-tuned models, an important detail when interpreting those comparisons.

These figures measure refusals, not every dimension of safety or the severity of every response. Nor do they imply that the uncompressed models were uniformly reliable: their baseline refusal rates differed considerably.

A proposed explanation

The authors propose that compression can retain a request’s semantic content while weakening internal features that help trigger a refusal. To investigate, they trained simple probes on decoder activations from uncompressed inputs, identifying directions associated with refusal. In the two ICAE-tested models, harmful prompts that originally triggered refusals showed much weaker projections along those directions after compression.

That observation is consistent with the proposed explanation, but it does not establish a complete causal account of how encoder training changes the decoder’s internal geometry. The preprint claims high reconstruction fidelity, yet it does not report a quantitative reconstruction evaluation alongside its refusal results. Preservation of the requests’ meaning should therefore be treated as an author claim rather than a verified finding of these experiments. The strongest direct evidence is the observed change in refusal behavior and the associated probe measurements.

A targeted repair, with boundaries

The authors also test Memory Steering, an intervention applied once to compressed memory before the model generates an answer. They derive a steering signal by comparing compressed prompts with versions containing an explicit safety-related cue. A separately trained gate estimates which memory slots carry unsafe intent, so the signal is not applied equally everywhere. The decoder itself is not retrained.

Steering raised harmful-prompt refusal rates in the tested ICAE systems. On HarmBench, Mistral-v0.2 rose from 16.5% without steering to 57.5% at the strongest tested setting, matching its uncompressed baseline. LLaMA-2 rose from 77% to 97.5%, compared with a 99% uncompressed baseline. On AdvBench, some steered rates exceeded their uncompressed counterparts. Ungated steering, however, caused substantial over-refusal of benign requests, especially for LLaMA-2.

For benign prompts, an automated judge rated most steered answers as tied with their unsteered compressed counterparts; depending on model and steering strength, 79.5% to 87.8% were judged tied or better. That is not proof of unchanged quality: some answers were judged worse, ties can include equally poor answers, and the comparison was against compressed output rather than uncompressed output.

The repair was evaluated for fixed-length ICAE memories, not variable-length ARC compression. The study also focuses on refusal rather than broader properties such as calibration or sensitivity to context. Its practical lesson is therefore a testable design warning, not a universal verdict on compression: efficiency transformations should be checked for the behaviors a system is expected to preserve, including safety-trained refusals.