Getting language models onto small devices takes more than shrinking them

The challenge beyond downloading a model

Running a language model locally could reduce dependence on remote servers, keep processing close to personal data and support applications without a reliable internet connection. But the work must then fit within the device’s memory, processing capacity and energy budget.

Rahul Maliakkal, Yashasvi Makin, Plawan Rath, Rohit Jain and Ankur Sadhoo examine these constraints in Large Language Model Deployment on Resource-Constrained Edge Devices: A Practitioner’s Survey. Its arXiv version was posted on November 6, 2026; the landing record also lists a related IEEE DOI. The arXiv posting alone should therefore not be read as establishing that the work is unpublished.

This is a review of existing techniques, not a questionnaire answered by practitioners or a new benchmark experiment. Its contribution is an integrated deployment guide: model compression, task adaptation, decoding software and hardware selection need to be considered together.

Three ways to make a model smaller

The survey organizes compression around quantization, pruning and knowledge distillation.

Quantization represents a model’s numerical values with fewer bits. Moving from higher-precision formats to 8-bit or 4-bit representations reduces storage requirements, but can introduce errors that affect answer quality. Post-training quantization applies this conversion to an already trained model. Quantization-aware training instead incorporates the effects of reduced precision during training, requiring additional training resources.

The authors discuss methods including GPTQ, SmoothQuant and activation-aware weight quantization, or AWQ. These address quantization errors in different ways; they are not presented as one interchangeable solution.

Pruning removes model components judged less important. Structured pruning removes larger units, such as neurons or layers, and is generally easier to accelerate on conventional hardware. Unstructured pruning removes individual weights, creating sparse patterns whose efficient execution can depend on specialized hardware or software. Fewer retained weights do not automatically translate into a corresponding speed improvement.

Knowledge distillation takes another route: training a smaller “student” model to reproduce aspects of a larger “teacher.” Rather than simply storing the same model more compactly, this produces a smaller model intended to preserve useful behavior. The survey describes combining distillation with other compression techniques.

Personalization is a separate problem

Using a model and adapting it require different resources. The authors treat training a large language model from scratch on an edge device as impractical, focusing instead on pretrained models and more economical adaptation.

Parameter-efficient fine-tuning changes a limited set of parameters or adds small trainable components rather than updating the entire model. LoRA learns compact updates while keeping the original model weights fixed. QLoRA combines this approach with a quantized base model.

These methods offer routes toward task-specific or local personalization. However, the survey does not demonstrate that every device capable of running a compressed model can also fine-tune it. Adaptation remains subject to memory and compute constraints.

Execution matters as much as compression

Generating text also involves repeated computation. Key-value caching retains information from earlier attention calculations so it can be reused during subsequent generation. Paged attention addresses how that cache occupies memory. Speculative decoding uses a smaller draft model to propose tokens for verification by the target model.

The survey connects these techniques to runtime software, including ONNX Runtime, ExecuTorch and llama.cpp, alongside serving engines such as vLLM and SGLang. It also considers CPUs, GPUs and dedicated AI accelerators. The practical point is that a compressed model still needs an execution system suited to its numerical format and target hardware.

A guide, not a device guarantee

The paper provides recommendations organized by device memory, alongside model examples, compression levels and power ranges. These are literature-based guidelines, not results from a controlled comparison conducted across those devices. Its reported savings and speedups should not be treated as universal guarantees.

The authors identify adaptive compression, more portable runtimes and lightweight continuous learning as future research directions. Those proposals are not demonstrated capabilities of this survey.

Its central lesson is practical: successful local deployment depends on balancing answer quality, memory, latency and energy together. Shrinking the model is only part of that decision.