SplitLite Shrinks the Data Chatter of Training Language Models

A language model’s internal activations are numerical representations of its input as it moves through the network. In split learning, the client processes part of the model and sends these representations to a server, which processes the rest. During training, gradients—the signals used to adjust model parameters—travel back across the same boundary. For large models, moving this information can become a substantial communication cost, particularly when devices have limited upload capacity.

SplitLite is designed for split federated fine-tuning with LoRA, a technique that updates small, low-rank adapter matrices while keeping the pretrained model’s main weights frozen. The authors’ central observation is that, for the same data sample, activations and gradients may change in structured ways between training epochs. Rather than transmit each full tensor again, SplitLite compares it with a cached version and sends a compressed representation of the difference.

Compressing changes, not entire tensors

The method uses truncated singular value decomposition, or SVD, to approximate each difference with a small number of important directions. A useful analogy is describing a complicated change using a few dominant patterns instead of listing every altered value. The resulting factors are then quantized, meaning their numerical precision is reduced to use fewer bits. The receiver reconstructs an approximate difference and adds it to its cache. The first time a sample is encountered, the full tensor is sent to establish that cache.

The paper measures how many singular directions are needed to account for 90% of the difference’s energy. In the reported examples, activation differences had effective ranks below the LoRA rank, while gradient differences stayed below twice that rank. This is evidence for the compression strategy in those settings—not a guarantee that all models or datasets will show the same structure.

What the experiments report

The researchers tested four pretrained language models on three GLUE sentence-pair tasks: MNLI, MRPC and RTE. Data was distributed across six clients in a non-identical way, and the main experiments used LoRA rank 16. The comparison methods included SplitLoRA without cut-layer compression, an activation-quantization approach, and a randomized sparsification method. The authors report averages and standard deviations over three runs for the main accuracy results.

Against SplitLoRA, SplitLite reduced activation uplink communication to 6.5%–9.6% of the baseline amount across the reported model-task combinations. Total communication fell to 11.4%–19.1% of the baseline. Those ranges correspond to the paper’s reported maximum reductions of 93.5% in activation uplink and 88.6% in total communication. These are relative results within the tested setup, not promises of the same savings on other networks or workloads.

The paper reports accuracy comparable to or higher than SplitLoRA in nearly all evaluated combinations, rather than a universal improvement. It also reports lower end-to-end training latency in its testbed experiments, despite the extra computation needed for SVD. Latency results depend on the authors’ particular hardware and communication assumptions, so they should not be read as a general speed guarantee.

What remains uncertain

The evidence is bounded by the study’s scope: four models, three GLUE tasks, six clients and a particular split-learning configuration. Results on these sentence-pair benchmarks do not show how SplitLite behaves on other language tasks, datasets, model sizes or real-world deployments. The method also maintains per-sample activation and gradient caches; the paper notes that storage can be substantial, especially for larger datasets. Compression requires choosing truncation ranks and quantization settings, which involves balancing communication against reconstruction error and model performance.

The authors provide a convergence analysis, but its guarantees rely on stated assumptions about smoothness, gradient variation and quantization error. Such theoretical conditions help describe when the analysis applies; they do not by themselves establish that every practical training setup satisfies them. The experiments also use a limited number of independent runs, so small reported accuracy differences should not be treated as decisive evidence of superiority.

SplitLite’s contribution is a targeted way to reduce repeated client-server traffic by exploiting temporal structure in LoRA fine-tuning. The results suggest that this can make split training more communication-efficient without an observed accuracy penalty on the tested tasks. Whether the same trade-off holds in broader applications remains an empirical question. This article is based on an arXiv preprint, which may not have undergone peer review; the paper and its claims should be read with that limitation in mind. Source: SplitLite: Low-Rank Residual Compression for Split Learning.