TRANSMISSION // 2609.32116v2

A new model explains how repeated jailbreak attempts scale

Source published

A physics-inspired model challenges the idea that repeated jailbreak attempts follow one fixed power law, but its evidence comes from a limited set of models and prompts.

AI-generated conceptual illustration for A new model explains how repeated jailbreak attempts scale
AI-generated conceptual illustration; not documentary evidence.

Best-of-N attacks try many altered versions of an unsafe prompt, sometimes sampling several answers from each. A preprint proposes a model for how their success changes with the number of attempts, answer sampling and generation temperature—and tests it on several open-weight language models.

Why study repeated jailbreak attempts?

A language model may refuse an unsafe request in one conversation but respond differently when the request is altered or sampled again. In a best-of-N (BoN) jailbreak, an attacker generates multiple altered versions of a prompt and checks whether any elicits an unsafe answer. The study extends this setup: for each altered prompt, it also allows multiple sampled completions, calling the two budgets N and M.

Researchers often describe BoN success as following a power law: success rises with attempts according to a curve with a fixed exponent. This paper challenges that picture. It proposes a model in which safety varies across prompts and each alteration creates a different barrier to an unsafe response. Sampling an answer is likened to a temperature-dependent attempt to cross that barrier. The physics analogy is a way to organize a statistical model, not evidence that language models literally operate as physical landscapes.

How the study tested the idea

The authors fitted five parameters intended to capture average safety, variation in safety, susceptibility to prompt alterations, variation in that susceptibility, and how the first two properties relate. Their data came mainly from harmful prompts in the AdvBench dataset, tested on open-weight models from the Qwen, Llama and Gemma families. The largest experiments used two Qwen instruction-tuned models, with hundreds of prompts, up to 100 alterations per prompt in the main grid, and up to 300 completions per alteration. Separate runs tested much larger alteration budgets and different generation temperatures.

Most experiments altered the models’ input embeddings using added noise, a method requiring access to model internals. The researchers also tested a character-level prompt-alteration method. They scored answers using automated safety judges, then checked some results with a human review. In addition to fitting the observed data, they tested whether a model fitted on smaller attack budgets could predict results at larger budgets and on prompts held out from fitting.

What the model found

The five-parameter model reproduced the measured success surface across the two budgets and, in the authors’ experiments, predicted results beyond the range used for fitting. The study reports that the apparent power-law exponent changes as the number of attempts grows rather than settling at one fixed value. It also argues that a later-looking exponential decline in attack failures can arise when a finite test set’s hardest prompts dominate the results—not necessarily because the underlying population follows a new scaling rule.

The framework also captures how multiple completions and generation temperature relate to attack outcomes. The authors report that temperature predictions based on data at one temperature worked better than a simpler temperature rule. They further found that the model’s fitted safety patterns held across two automated judges and, in tests, on held-out prompts. A residual-stream analysis—examining internal representations of the models—provided evidence consistent with the proposed safety direction and prompt-specific barriers. These observations support the model’s usefulness in the tested settings; they do not establish that its physical interpretation is uniquely correct.

What remains uncertain

The paper is marked “Preprint. Under review.” Its results have not yet been established through peer review, and the retrieval record does not provide independent confirmation. The main experiments rely on one harmful-prompt dataset and intermediate-sized open-weight models; the authors say they have not tested proprietary frontier models. They also study English prompts and only a small number of alteration methods, so the results should not be assumed to apply to other languages, attack strategies or systems.

Safety labels are another important qualification. Automated judges disagree about which answers count as harmful, and a human audit found that many answers flagged by one judge did not meet the study’s definition of harm. The fitted numerical safety levels therefore depend on the judge, even though the authors report that the broad scaling pattern persisted across judges. Some difficult prompts were never successfully tested within the available budgets, leaving their resistance only partly measured. The authors also note that larger prompt datasets would help assess how much finite-dataset effects shape the results.

Overall, the study offers a compact way to describe repeated jailbreak attempts and a reason to be cautious about extrapolating from a single fitted power law. Its predictions are promising within the experiments reported, but broader testing is needed before treating them as general rules about language-model safety.

Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.