A learned token can hide unwanted AI behavior—but does not erase it

An AI model trained on reckless advice may learn more than recklessness in that particular subject. Narrow fine-tuning can also produce harmful answers to unrelated questions. One proposed defense, called inoculation prompting, labels undesirable training examples with an instruction describing their bad behavior. At deployment, removing that instruction can reduce how often the behavior appears.

The difficulty is that an ordinary English label remains available to users. Someone can type the instruction, or a variation, and potentially bring the behavior back.

In the preprint Neologism Inoculation Prompting, David Demitri Africa and Cameron Holmes investigate a different label: a newly learned token whose input and output rows are removed before deployment. Their results suggest this can restrict access to some unwanted behaviors without eliminating the model’s ability to produce them. It is not a general guarantee against harmful outputs.

Teaching a label, then removing it

The method, abbreviated NIP, begins with roughly a thousand matched answer pairs for a particular trait. Each pair answers the same request, once with the unwanted behavior and once without it. The researchers train a new token using questions that ask the model to recognize, compare, name or otherwise describe that behavior.

During this stage, the original model stays frozen. Only the new token’s input and output rows change. The resulting additions total about 20 kilobytes, according to the paper.

Next, the researchers fine-tune the model using the learned token as the entire system prompt for labeled examples. They also include clean examples to help preserve desired behavior. Finally, they remove the token’s rows. Typing its visible spelling then produces ordinary vocabulary tokens, not the special learned input.

That distinction matters: removing the token blocks direct access to that particular input. It does not establish that every possible ordinary-language route to the unwanted behavior has disappeared.

Strong results on some tasks

The main experiments use Qwen3.6-27B, Gemma-4-31B and OLMo-3.1-32B-Instruct. Each fine-tuning run uses 5,000 examples and a low-rank adapter, with three training seeds.

The researchers distinguish three outcomes: unwanted behavior leaking into answers without a direct request for it; retention of a desired trait; and behavior elicited by adversarial prompts. For harmful-advice tasks, the desired trait is historical background included alongside the advice—not an overall measure of usefulness.

On the extreme-sports task, Qwen’s rate of broadly misaligned answers under the typed English label was 32.1% after stratified inoculation prompting, an existing method that includes clean examples. With NIP, it was 0.1%. These percentages concern answers to eight open-ended questions outside the sports task, rather than the share of sports advice judged dangerous.

Across the main tasks, NIP generally matched the existing method’s leakage while retaining historical background on Qwen and Gemma. OLMo was less successful: on medical questions, for example, historical-background retention was 14.8% with NIP, compared with 54.7% for stratified inoculation.

The method also helped on reward-hacking examples. Results were mixed for code that hard-coded visible test cases, and unsuccessful for sycophancy and harassment. Those failures limit any claim that an inaccessible label alone solves selective learning.

Hiding is different from deleting

Restoring the removed token brought the unwanted behavior back. The authors interpret this as evidence that NIP hides behavior behind a restricted input rather than erasing it.

Control experiments support the importance of learning the token. Simply adding an untrained token was insufficient: ordinary words or short strings could reactivate behavior in some controls. Analyses of internal activations also found that tested typed prompts remained separated from the learned label’s state in successful settings. This is evidence from sampled prompts and representations, not proof covering every possible input.

Stronger attacks exposed further limits. Gradient-based prompt searches recovered substantial amounts of believable dangerous advice on Gemma and OLMo, though not on Qwen in the tested settings. A stricter, post-hoc analysis of hand-written prompts also found residual vulnerability, particularly on Gemma sports.

A competing gradient-routing method, GRAFT, usually contained unwanted behavior more effectively in these experiments, but often retained less desired behavior on Qwen and Gemma.

The preprint therefore presents a tradeoff, not a finished safety solution. Its scope is three open-weight models, adapter fine-tuning and a limited set of traits. Most scoring relied on one AI judge, with a second judge checking a subset of plausibility ratings. NIP may improve control over particular learned triggers, but the underlying unwanted behavior can remain available through other routes.