TopoQuant probes how compression changes language models’ attention

Looking beyond benchmark scores

Post-training quantization reduces the numerical precision used to represent a trained language model’s parameters, lowering deployment costs without retraining the model from scratch. The tradeoff is that rounding and compression can change its behavior. Aggregate accuracy alone does not reveal whether a compressed model makes the same decisions on individual examples.

TopoQuant examines another aspect of preservation: whether quantized models attend to the same context tokens as their base models. The authors propose using topological data analysis, a mathematical approach to studying connectivity and shape, to connect changes in attention-weight structure with changes in attention allocation.

This distinction matters throughout the study. Preserving attention patterns is the measured target; it is not the same as proving that generated answers, reasoning, or overall task performance remain unchanged.

How the framework works

The pipeline extracts the Query, Key, and Value projection matrices from each transformer block in both the base and quantized models. These components participate in forming attention and carrying information through it. TopoQuant analyzes them separately rather than combining all attention parameters into one score.

For each weight matrix, the researchers calculate pairwise cosine distances and use the Ripser software to construct a persistence diagram. Their implementation tracks connected components as a distance threshold increases, with a filtration threshold of 0.8. In accessible terms, it records how groups connect as the criterion for treating them as neighbors becomes less restrictive.

Seven measures compare the base and quantized diagrams, including persistence entropy, persistence landscapes, and sliced Wasserstein distance. The resulting differences quantify what the paper calls topological drift.

To measure attention consistency, the researchers run matching prompts through both models. They examine the final query token’s attention, excluding attention to itself and special tokens. Top-1 agreement checks whether the most-attended context position matches. Top-3 intersection over union measures the overlap between the three most-attended token sets, divided by their combined set size.

What the experiments found

The evaluation covers four models: Llama-3.2-3B, Vicuna-7B-v1.5, Mistral-7B-v0.1, and Llama-3.1-8B. Quantized variants were created with llama.cpp using legacy and K-quantization formats spanning nominal two- to eight-bit settings. Attention comparisons used 500 samples from each of ARC, HellaSwag, and WinoGrande.

The authors report relatively limited topological drift at moderate compression, followed by sharper increases in the sub-four-bit configurations. These increases accompany declining attention agreement, particularly for Query and Key projections.

The tables also show that preservation varies substantially by model and benchmark. On ARC, for example, Mistral-7B’s Top-3 overlap falls from 0.99 under Q8_0 to 0.66 under Q2_K. Llama-3.1-8B falls from 0.74 to 0.58. Thus, limited drift at higher precision should not be read as near-perfect attention agreement for every model.

Predicting attention agreement

The researchers train linear regression, random forest, and XGBoost models using five-fold cross-validation. Inputs combine topological measurements with configuration information, such as quantization settings and block category. The prediction target is average Top-3 attention overlap.

The strongest reported results reach R² values of 0.85 on HellaSwag and 0.83 on ARC and WinoGrande. R² describes how well predictions explain variation in the measured target; it is not an accuracy percentage. These headline results use combined features, not topology alone. Standalone topological features perform considerably less well in the reported ablations.

The authors interpret the combined results as evidence that structural measurements could help screen compressed checkpoints before extensive benchmarking. Their runtime comparison for Llama-3.1-8B reports 862–1,250 seconds for TopoQuant calculations, including attention agreement, versus 4,603–4,709 seconds for downstream ARC evaluation. That comparison is specific to the reported setup.

Boundaries of the evidence

Value projections show less drift, but the tested K-quantization formats give them higher precision than Query and Key projections. The authors acknowledge that this prevents attributing the difference solely to intrinsic robustness.

The study examines attention projections rather than the full network. Other quantization approaches, including AWQ and SmoothQuant, remain future work, as does optimization for models exceeding 70 billion parameters. Although weight-based topological measurements do not require evaluation prompts, the attention measurements used to train and assess the predictors do.

TopoQuant therefore offers evidence for a potentially useful compression diagnostic within the tested settings. Its proposed early-warning role should be distinguished from a demonstrated universal stopping rule or a replacement for all downstream evaluation.