FP4Quant speeds up GPU communication, with a measurable accuracy trade-off

Training a language model across several GPUs requires more than dividing up the calculations. The devices must repeatedly exchange intermediate tensors: arrays of numbers that connect their portions of the computation. Compressing those messages can reduce communication costs, but every discarded bit risks changing the information used during training.

In the FP4Quant preprint, Man Liu and colleagues propose a framework for compressing model-parallel communication into four-bit floating-point values. Their experiments report higher training throughput in tensor-parallel settings, alongside an important qualification: the fastest configuration is not the one that best preserves model quality.

Why four bits need more than rounding

The paper considers tensor parallelism, which splits individual model layers across GPUs, and pipeline parallelism, which assigns different layers to successive stages. Tensor parallelism requires frequent collective communication, while pipeline stages exchange activations and gradients.

Four-bit floating-point values offer a much smaller representation than the 16-bit BF16 baseline. However, their restricted numerical range and coarse spacing make them vulnerable to outliers and rounding errors. A few large values can determine a shared scale, leaving smaller values poorly represented.

FP4Quant addresses this by splitting each communication tensor into small, contiguous groups. It first applies a group-specific scale based on the root-mean-square magnitude. After a mathematical transformation, a second scale uses the largest absolute transformed value to fit the group into the available four-bit range. Both scales accompany the compressed values for reconstruction.

The transformation is a normalized Walsh–Hadamard transform, implemented through repeated additions and subtractions. Its purpose is to spread concentrated values across coordinates before quantization. The transform itself is reversible; the subsequent four-bit rounding is not.

Making compression cheap enough to help

Reducing message size is useful only if compression and decompression do not consume the time saved. FP4Quant therefore also targets the mechanics of GPU execution.

Within a GPU warp, a group of cooperating threads, the transform exchanges values through register-level shuffle instructions. Larger groups require some exchanges through shared memory and synchronization. Separating these stages is intended to limit that overhead.

Storage presents another problem: a four-bit value occupies only half a byte. FP4Quant pairs neighboring threads’ codes and assembles two codes into each byte directly in registers, avoiding intermediate memory round trips for packing. At the receiver, unpacking, dequantization and the inverse transform reconstruct an approximation of the original tensor.

This is a communication-compression method, not evidence that every computation or model component runs in four-bit precision.

The speed–quality trade-off in the results

The researchers evaluated Qwen-7B training on OpenWebMath using a single machine with eight NVIDIA H100 GPUs connected through NVLink. Comparisons included uncompressed BF16 communication and Tahquant, a method developed for pipeline-parallel activation compression. Reported convergence curves cover 10,000 iterations.

For the table’s four-way tensor-parallel configuration, BF16 achieved 111.0 TFLOP/s per GPU with validation loss of 2.1505. FP4Quant with groups of 64 elements reached 155.3 TFLOP/s per GPU, a reported 1.40× speedup, while loss increased to 2.2306. Lower loss is better.

The headline 1.56× speedup came with groups of 256 elements, reaching 173.3 TFLOP/s per GPU. But validation loss rose to 3.0616, and perplexity increased from the BF16 baseline’s 8.5893 to 21.3613. The maximum speedup therefore should not be read as an accuracy-preserving result.

Tahquant was faster still in this tensor-parallel comparison, at 1.78× the baseline throughput, but its validation loss was worse at 3.4848. FP4Quant offered a different trade-off rather than winning every metric.

Pipeline parallelism showed a different pattern. With groups of 64, FP4Quant’s loss was 2.1525, close to BF16’s 2.1505, while throughput increased from 189.1 to 190.1 TFLOP/s per GPU. Groups of 512 sharply worsened loss in both parallel regimes.

What remains unproven

Downstream evaluations with groups of 64 also show that quality preservation depends on the setting. Under tensor parallelism, C-Eval validation accuracy was 48.29%, versus 52.15% for BF16. Pipeline and combined model-parallel configurations produced C-Eval point estimates slightly above the baseline; those numbers alone do not establish a reliable improvement.

The evidence is limited to one model, one training corpus and a single eight-GPU node. Larger models, multi-node scaling and adaptive precision are proposed future investigations, not demonstrated results. The preprint’s practical message is narrower than lossless acceleration: four-bit communication can improve throughput, but group size and parallel configuration materially affect the quality cost.