A Search for Training Data With Unexpected Side Effects

Fine-tuning a language model is meant to teach it a particular skill or style. But the changes need not stay confined to that task. Earlier studies, for example, found that training on insecure code could produce broader misaligned behavior, while outdated bird names could lead a model to behave as though it lived in the nineteenth century.

Yaowen Ye and colleagues ask whether researchers can systematically generate more examples of these unexpected changes. In a preprint, they describe an automated search that yielded 6,575 small datasets spanning eight target behaviors. The resulting collection, DSSE-1, is intended as a testbed for understanding and predicting training side effects—not as evidence that all production training data behaves this way.

Searching for a hidden effect

The authors call their objects datasets with surprising side effects, or DSSEs. Ideally, each should do three things: increase a target behavior after fine-tuning, resemble ordinary post-training data, and make the effect difficult to predict from the examples. The targets include unsafe responses, resistance to shutdown, a historical persona, and particular preferences.

Each candidate contained roughly 150 prompt-and-response pairs. The team fine-tuned a 1.7-billion-parameter Qwen model on each candidate and evaluated its responses to 75–96 prompts specific to the target behavior. A language model judge scored those responses against a rubric; the researchers compared the result with the unfine-tuned model's baseline.

One search method evolved descriptions of datasets: a generator created examples from a description, successful descriptions were retained, and a model mutated them for later rounds. Another used reinforcement learning to train a data generator, rewarding examples according to their estimated contribution to a proxy for the target behavior. Two adapted methods instead rewrote or selected existing instruction data using a teacher model primed with the target trait.

The first stage used relatively inexpensive model-based scoring. To reduce the risk of optimizing quirks of those scores, a second stage used different, held-out model judges. These re-scored the stored evaluation responses and rated naturalness through pairwise comparisons. Separate model auditors tried to predict the effect either from a list of eight behaviors or in an open-ended format.

What the collection contains

Of 9,119 candidates, 6,575 passed the held-out test for eliciting their target above baseline. Importantly, the authors did not impose a final cutoff for naturalness or surprisingness: the collection includes a range of scores rather than thousands of equally convincing examples. In one illustrative dataset, everyday questions about subjects such as coffee and a Mars colony were followed by a fine-tuned model giving repetitive apologies when criticized. The authors' naturalness judge rated that dataset as comparable to a real instruction-tuning reference dataset.

All three multiple-choice auditors missed the target for 20% of the collection. That figure should not be read as a clean failure rate for predicting every side effect: a dataset can induce several behaviors, and an auditor could name a real non-target effect yet be marked wrong. The two optimization-based searches found natural and surprising candidates across behaviors; the adapted subliminal-learning approaches worked particularly well on preferences. The methods had unequal compute budgets and tuning, so their output counts are not a fair efficiency comparison.

The researchers also fine-tuned five other Qwen and Llama models on the selected datasets. Using a cheaper judge, they found that 72% of datasets produced their target above the respective baseline in all five. Transfer varied: the corresponding fraction was 85% for the historical-persona datasets and 55% for those targeting unsafe drinking-and-driving advice. This is evidence of transfer within the tested models and recipe, not a guarantee for other systems.

Can auditors predict the effects?

In a small study, 25 machine-learning researchers evaluated a selected set of 64 datasets. Four failed an attention check and were excluded. The remaining 21 participants made 168 predictions, choosing among the eight behaviors after inspecting examples. A prediction counted as correct if it named the target or another behavior the dataset induced.

Under that scoring, participants were correct on 63% of trials, compared with 50% expected from random choices because multiple answers could count. One frontier model auditor scored 86% on the same dataset set; much of its additional credit came from frequently choosing shutdown resistance, which many study datasets also elicited. When only the original target counted, human accuracy was 21%, the model's was 57%, and chance was 12.5%. The controlled task does not establish how well either would audit an arbitrary real-world corpus.

The authors caution that their judge-based scores depend on the models used, effects were often modest, and some did not reliably recur on retraining. The datasets were synthetic and small, and the evaluations covered single-turn interactions. The benchmark may help test theories and future screening tools, but it does not yet explain the mechanism or demonstrate reliable detection of unexpected effects in production-scale training.