A Head-by-Head Approach to Faster AI Attention

Why selection needs its own strategy

Attention lets a language model combine information from different positions in its context. A query represents what the model is looking for, keys help determine which positions are relevant, and values supply the information to combine. Dense attention compares a query with every available preceding key, creating substantial computation and memory traffic as context grows.

Sparse attention reduces that work by consulting only selected positions. But finding those positions has a cost. Large blocks are cheap to assess, yet can bundle useful tokens with irrelevant ones. Small blocks offer more precise selection but require more candidates to be scored.

In the preprint Attend at the Right Resolution, Wajih Hassan Raza and colleagues propose Head-Adaptive Sparse Attention, or HASA. Their central idea is that selection should reflect the interaction structure of each pretrained attention head rather than use the same resolution throughout the model.

A compact guide, not a replacement memory

HASA adapts selection along two dimensions: the features used to estimate relevance and the sizes of neighboring context regions.

First, the researchers combine each head’s pretrained query and key transformations into a mathematical operator describing their interaction. They analyze it using singular value decomposition, which identifies paired feature directions and their associated strengths. Some heads concentrate their interaction strength in relatively few directions; others spread it more broadly.

HASA chooses how many directions to retain separately for each head. It combines spectral entropy, a measure of how widely those strengths are distributed, with a constraint on the spectral energy discarded. This analysis happens offline, without additional model training, producing compact representations for estimating relevance.

Second, HASA uses those representations to organize the sequence. Starting with contiguous eight-token regions, it progressively merges neighbors whose token responses point in sufficiently similar directions, subject to a maximum region size. Importantly, it assesses agreement across the token directions, not merely similarity between two regional averages: similar averages can conceal substantial internal differences.

Each candidate region has a mean response and a measure of dispersion. A query scores these summaries, selects high-ranking regions, and also includes local context. The model then retrieves the original, full-dimensional keys and values at the selected positions.

That distinction matters: compact representations guide selection, but do not replace the original information used for the final attention calculation. Attention is exact over the selected positions—not equivalent to dense attention over the entire context.

What the experiments found

The authors evaluated Ministral-3-8B-Instruct-2512 and Qwen3-8B on LongBench-Pro, covering English and Chinese long-context tasks, and RULER-32K, a controlled synthetic benchmark. Sparse methods received matched decoding budgets of 512, 1,024 or 2,048 key-value tokens per query.

On Ministral at the smallest budget, HASA scored 37.43 overall on LongBench-Pro, compared with 36.58 for the closely related UNIQUE method and 37.70 for dense attention. On RULER-32K, the corresponding averages were 91.16, 90.78 and 93.15. HASA also had the highest overall scores among the sparse methods evaluated on Qwen3-8B at each tested budget, although it did not win every individual category.

The distinction between overall and task-specific gains is important. At the 512-token Ministral budget, the reported 3.47-point RULER improvement over UNIQUE appears in the CWE task, while the 2.57-point LongBench-Pro improvement appears in the Moderate difficulty category. These are not gains in the benchmarks’ overall scores.

In a separate selection analysis, HASA retained 88.1% of dense-attention mass—the attention weight assigned to positions it selected—versus 87.5% for UNIQUE and 80.7% for a fixed-resolution reference. An ablation study also found that combining feature and sequence adaptation performed better than either alone.

Reported self-attention speedups over dense attention increased from 6.4 times at an 8K context to 14.07 times at 32K. Against UNIQUE, the 32K speedup was 1.32 times. These measurements concern self-attention, not end-to-end model execution.

Where the evidence stops

HASA’s offline characterization does not capture every factor governing attention at runtime. The authors specifically identify softmax normalization, value representations and downstream computation as missing from that characterization.

The theoretical analysis also has conditions: preserving the selected regions under feature truncation requires sufficiently separated relevance scores and a fixed collection of candidate regions. It does not guarantee identical region partitions or full-model outputs.

The results support a targeted conclusion from this preprint: tailoring selection resolution to individual heads improved the tested accuracy–efficiency trade-off. They do not establish universal speedups or lossless replacement of dense attention.