Compressing Small Language Models: When Combining Techniques Helps—and When It Fails

Three ways to shrink a model

A language model with seven billion parameters needs roughly 14 GB for its weights in half-precision format, according to this preprint. That creates pressure to reduce memory requirements before deployment on constrained hardware.

The authors examine three interventions. Quantization stores weights at lower numerical precision. Pruning sets selected weights to zero; here, the researchers used unstructured magnitude pruning, targeting smaller-magnitude weights. Recovery training adjusts a compressed model afterward to repair lost performance.

Their central question is how these interventions interact, rather than whether each works alone.

A controlled comparison across four models

The study covers Phi-2, Qwen3-4B, Nemotron-Mini-4B and Qwen2.5-7B-Instruct, ranging from 2.7 billion to seven billion parameters. The main experiment combines three precision settings, three pruning settings and recovery either enabled or disabled. This produces 72 configurations, with quantization followed by pruning and then recovery. Another 48 runs examine alternative orderings.

The main quality measure is WikiText-2 perplexity, which measures how well a model predicts text; lower values indicate better prediction on that dataset. The authors also report accuracy evaluations on MMLU, HellaSwag, ARC-Challenge and BoolQ for 30 compressed configurations.

GPU experiments used one NVIDIA L40S. Separate CPU measurements used llama.cpp to examine whether compressed models delivered practical memory and throughput benefits.

Quantization is useful, but order can decide survival

The authors report that INT8 quantization alone reduced GPU memory use by 40–45%, with less than 0.5% average benchmark accuracy degradation. Its perplexity remained within one point of the half-precision baseline across the four models. Four-bit NF4 quantization without pruning also remained usable, although benchmark accuracy losses were larger.

The sharpest failure appeared when NF4 quantization preceded pruning. All 16 such combinations in the main experiment exceeded the study’s collapse threshold of 1,000 perplexity, including those given recovery training. No INT8 configuration collapsed at either tested pruning level.

This is not evidence that NF4 and pruning are universally incompatible. In the ordering experiment, pruning before quantization produced viable NF4 models, with reported perplexity around 20–21. The failure therefore depends on the tested sequence and methods.

Among viable configurations, the authors found no statistically significant ordering effect on perplexity, reporting an ANOVA p-value of 0.96. That result concerns quality among surviving configurations; it does not erase the importance of ordering for avoiding collapse.

Recovery did not consistently need a teacher

Recovery used low-rank adaptation, or LoRA, which trains small added parameter matrices rather than updating the entire model. The researchers trained these adapters for 200 steps on 256 C4 text samples.

For the half-precision seven-billion-parameter Qwen model with 50% pruning, the short evaluation showed perplexity falling from about 921 to 24 after recovery with self-distillation. That is a roughly 97% reduction in degraded perplexity, not a 97% gain in accuracy or complete restoration to baseline.

A separate comparison tested whether a teacher model was necessary. On full-corpus evaluation, ordinary LoRA fine-tuning without teacher outputs performed better for two models and worse for two. The authors interpret this mixed result as evidence that LoRA adaptation, rather than the teacher signal, drove recovery in these experiments.

Zero weights do not automatically save resources

On CPU, the authors report that llama.cpp’s four-bit formats generated tokens 46–54% faster than its eight-bit formats across the four models. Within each model, the perplexity difference was under half a point.

Unstructured pruning delivered no CPU throughput or memory advantage. With dense storage and computation, zeroed weights still occupy space and do not automatically reduce execution work. Practical pruning benefits therefore require an execution system that exploits the relevant sparsity pattern.

Where the findings stop

These results support INT8 alone as the authors’ simplest GPU deployment recommendation, not a universal compression rule. Only four models and one CPU machine class were tested; mobile ARM and NPU hardware were not evaluated.

The CPU experiments used GGUF quantization rather than the GPU study’s bitsandbytes formats, and absolute perplexity values should not be compared across engines. The full sweep used one seed, with three-seed checks limited to 16 key configurations. All accuracy benchmarks were English-only. The evidence favors testing the exact model, compression order and runtime together before assuming that a smaller numerical representation will preserve quality or improve deployment performance.