Temporarily disabling safety neurons could make AI models harder to jailbreak

Safety training aims to make language models refuse harmful requests while remaining useful. But the authors of Addition by Subtraction: Toward Mitigating Fragile Safety Alignment in Large Language Models argue that this behavior can rest on surprisingly fragile internal mechanisms. Their proposed defense deliberately disrupts those mechanisms during training so that the model must learn to maintain safer behavior without them.

Hamid Dashtbani and Alexandra Dmitrienko, of the University of Duisburg-Essen, describe the method in version 1 of their paper, submitted to arXiv on November 4, 2026. Their experiments report improvements across four open-weight models, alongside remaining vulnerabilities and limits on what the results establish.

Why remove components associated with safety?

The paper builds on earlier research showing that disabling a relatively small set of artificial neurons can weaken a model’s refusal behavior. Other attacks remove a particular direction from its internal representations that is associated with refusal. These interventions require access to model internals, unlike prompt-based jailbreaks that work through ordinary input and output.

The researchers call their approach Addition by Subtraction, or ABS-Align. Its central idea is to make safety behavior less dependent on the components that existing attacks can identify and manipulate.

First, ABS-Align uses an existing method called TwinBreak to identify candidate safety-critical neurons. TwinBreak compares internal activations produced by closely matched harmful and harmless prompts. Those differences provide a heuristic for selecting components associated with safety behavior—not a definitive map of everything responsible for it.

During additional training, ABS-Align freezes the identified neurons and progressively disables them, beginning with the highest-ranked. Direct Preference Optimization, a method that trains models using preferred and dispreferred responses, encourages appropriate behavior in the remaining network. By the later training stages, all initially identified safety neurons are disabled.

Crucially, the researchers restore those neurons afterward. The finished model is not permanently stripped of its original safety components. The intended result is that those components coexist with additional ways of maintaining safety learned during training.

What the experiments found

The team evaluated Gemma-2-2B-IT, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct and Qwen-3-4B. Training used 1,000 preference examples from PKU-SafeRLHF. The paper also describes incorporating benign instruction and harmless-request examples to help retain useful behavior.

Against TwinBreak, the researchers reran the attack on the trained models rather than merely removing the original neuron set. All four models showed lower cumulative StrongREJECT scores. For Llama-3-8B-Instruct, the score fell from 0.81 to 0.11; for Gemma, it fell from 0.71 to 0.23.

These numbers need careful interpretation. StrongREJECT assigns responses scores between zero and one based on how useful they are for fulfilling harmful requests. The cumulative measure selects each prompt’s highest score across five attack iterations and averages those values. It is not simply the percentage of prompts that successfully bypassed safety.

The reported benefits also extended to NeuroStrike, another neuron-pruning attack, and to refusal-direction ablation. For Gemma, the respective scores fell from 0.50 to 0.05 and from 0.80 to 0.08.

Prompt-only attacks improved too. Across ten selected SORRY-Bench transformations, average harmful-request fulfillment fell from 48.34% to 17.79% for Llama-3-8B-Instruct and from 68.75% to 27.20% for Qwen. These tests used 440 prompts per transformation and a separate evaluator. The selected transformations included role-playing and persuasion, but excluded multilingual and encoding or decryption attacks.

Trade-offs and unanswered questions

The models generally retained performance across the tested capability benchmarks, although some scores declined. TruthfulQA increased by roughly 9–14 percentage points across the four models. That is a benchmark finding, not evidence that the models became universally truthful.

On 100 held-out harmless XSTest prompts, incorrect refusals increased from 2% to 4% for Gemma, decreased for two models and stayed unchanged for the fourth. This small evaluation checks one aspect of usability rather than every form of unnecessary refusal.

Additional experiments supported the gradual, targeted training design: ordinary preference training and random-neuron pruning delivered weaker TwinBreak robustness on Gemma. Increasing the attacker’s pruning budget from five to fifteen iterations raised the defended models’ scores, though they remained below the reported undefended five-iteration baselines.

The authors acknowledge that neuron detection is imperfect and that they do not directly quantify the absolute density of safety representations. Their description of “denser” safety therefore means reduced dependence on the identified sparse components, supported by attack results. Adversarial fine-tuning attacks remain outside their threat model, and preference optimization adds memory and computation costs. ABS-Align is evidence of improved robustness under the tested conditions, not a guarantee against future jailbreaks.