Context-aware steering gives language models a more flexible control dial

A language model rewriting a document might need to make one passage negative, another positive, and leave a third unchanged. Applying the same internal nudge throughout the response is poorly suited to that assignment. An arXiv preprint by Brandon Hsu, Daniel Beaglehole, Adityanarayanan Radhakrishnan and Mikhail Belkin proposes a more flexible approach: retain a fixed steering direction, but learn how strongly to apply it as the context changes.

Their method, Contextual Linear Activation Steering, or CLAS, improves on fixed-strength steering across many of the reported experiments. These are findings from a preprint, not established guarantees of reliable model control.

Separating direction from strength

Activation steering changes a model’s intermediate numerical representations during generation. A steering direction is a vector associated with a concept or behavior, such as positive sentiment. Conventional linear activation steering adds that vector to the model’s activations after multiplying it by a fixed coefficient.

The approach draws on the linear representation hypothesis: that some semantic concepts correspond to directions in a model’s activation space. That hypothesis motivates the method; it does not imply that every behavior has a simple, fully understood internal representation.

CLAS separates two jobs. First, a probe learns to distinguish prompts with and without concept-related information, providing a steering direction for each transformer block. Those directions remain fixed. Second, a learned “sensing vector” computes the steering coefficient from the current activation, with an additional bias term. The strength can therefore vary by block and token position, rather than remaining constant throughout a response.

The underlying model parameters stay frozen. Only the sensing vectors are trained to predict desired completion tokens. This training uses example prompt-completion pairs rather than repeatedly generating responses to search for a suitable fixed coefficient.

Gains under limited supervision

The researchers tested 11 tasks on four instruction-tuned models: Qwen2.5-7B, Llama-3.1-70B, Llama-3.1-8B and Llama-3.2-1B. Tasks included mathematical problem solving, translation, sentiment rewriting, summarization, safety refusal and answering unsafe requests.

The main setup used ten training prompt-completion pairs per task, plus ten validation pairs. CLAS also needed a separate labeled probe dataset—usually 200 training examples, with smaller datasets for some tasks. Its results should therefore not be described as learning from only ten labeled examples overall.

Using directions extracted by Recursive Feature Machines, CLAS achieved an average task score of 86.5% on Qwen2.5-7B, compared with 65.1% for fixed-strength steering. On Llama-3.1-70B, the corresponding averages were 88.0% and 74.4%. These percentages aggregate task-specific scores, some assigned by GPT-5 mini and others checked through exact matching; they are not a single universal measure of intelligence.

The gains were uneven. Averaged across models, mathematical problem-solving scores rose substantially, while complex-vocabulary steering performed worse than the fixed-strength baseline. Against rank-one LoRA and ReFT adaptations, CLAS led the ten-task average excluding JailbreakBench on three models, but both alternatives scored higher on the smallest model.

Coefficient tuning was also faster in the reported setup. For Llama-3.1-8B, CLAS averaged 195 seconds versus 3,910 seconds for fixed-coefficient grid search. This comparison concerns coefficient tuning, not the entire workflow including steering-direction extraction.

Different interventions within one response

A separate experiment required JSON output containing two edited versions of a document and a third field copying the original. The desired edits differed between fields—for example, negative sentiment in the first and positive sentiment in the second.

The authors report that CLAS achieved high scores on both edited fields while preserving the required schema and nearly preserving the copied text. The tested fixed-coefficient approach did not achieve both behaviors together. This demonstrates selective steering in that experimental setting, rather than proving that every alternative steering design must fail.

Interpretability and side effects

The probe-derived directions also detected labeled concepts more accurately across model blocks than directions learned by the tested LoRA and ReFT adapters. That supports their usefulness as concept monitors, but does not establish a complete explanation of the model’s reasoning.

Steering still changed performance on unrelated tasks, sometimes substantially. The experiments used 50 held-out prompts per task, greedy decoding and bounded response lengths. Some training completions were model-generated, and many evaluations relied on a model judge. The inclusion of both refusal and anti-refusal tasks also shows that stronger behavioral control is not inherently safer.

The central contribution is narrower and concrete: keeping an identifiable steering direction while learning a context-sensitive strength can make that intervention more effective and selective. Whether it remains reliable beyond these models, datasets and evaluation conditions requires further testing.