CHARM+ rearranges GPU communication to speed up distributed attention

Long-context language models face a coordination problem as well as a computation problem. When a sequence is divided across GPUs, each processor needs information held elsewhere to complete attention. Fast processors can therefore spend time waiting for transfers instead of calculating.

In the CHARM+ paper, available on arXiv and accepted to NeurIPS 2026 as a poster, Xinhang Zhang and colleagues at Lenovo Research propose a different communication schedule for distributed attention. Their approach combines a merge-based exchange structure, independent communication streams and a strategy for transfers across hardware boundaries. The authors report improvements in attention benchmarks and full training experiments, with benefits that vary across systems.

Why the ring creates waiting

Ring-Attention passes blocks of keys and values between GPUs arranged in a logical ring. Each GPU uses incoming blocks to update its attention result. Communication can overlap with computation, but successive transfers have dependencies: one exchange must finish before the next can proceed.

Transfers within a node or NUMA domain can be faster than transfers across those boundaries. Synchronizing the two can leave faster local communication waiting for slower exchanges elsewhere.

CHARM+ aims to reduce the communication time left exposed after overlap with computation—the periods when a GPU has no useful calculation to perform.

Three changes to the schedule

The first change replaces the linear exchange structure with a hierarchy of merge levels. With eight GPUs, for example, the basic merge mechanism has three levels. Groups exchange and aggregate key and value blocks, creating more computation to overlap with subsequent communication.

The authors describe this as reducing the communication dependency structure from a linear number of steps to logarithmic levels. That concerns scheduling dependencies, not a reduction of the underlying attention computation to logarithmic complexity.

The second change is the Re-Merge Multistream Mechanism, or RM+. Instead of communicating only keys and values, it also sends queries through independent streams. A GPU can process another GPU’s queries using its local keys and values, then return the resulting attention contribution. This creates additional opportunities to overlap transfers and calculations.

The third change addresses unequal bandwidth across hardware boundaries. Cross-Hardware Query Parallelism separates cross-node or cross-NUMA exchanges from communication inside each hardware group. Queries arriving from other groups are processed through local pipelines, with intermediate cross-hardware transfers overlapped where possible. This reduces repeated synchronization between fast local links and slower external links.

What the measurements show

The evaluation covers L20, H20 and H100 systems, measuring both forward attention computation and the backward computation used during training. Configurations are measured over 100 iterations after 20 warm-up iterations. Comparisons include Ring-Attention and TASP, although TASP is evaluated only for forward performance because its backward implementation was unavailable.

There is a verified discrepancy between the exact arXiv v1 landing page and the v1 PDF. The landing page’s displayed abstract and citation metadata both report a 2.9× average backward speedup over Ring-Attention on a dual-node NVLink architecture; the PDF abstract reports 3.9×. Both report 2.8× for forward computation. The supplied landing-page abstract metadata remains unchanged; the conflicting backward averages are left unreconciled.

A specific configuration using 32 H100 GPUs across four nodes, a batch size of 128 and a sequence length of 3,968 achieves reported forward and backward speedups of 2.69× and 4.21× over Ring-Attention. These are attention measurements, not whole-model training speedups or a resolution of the abstract discrepancy.

For full training, the researchers integrate CHARM+ into Megatron-LM and compare it with Megatron’s ring implementation. Across four GPT3-350M configurations on 16 GPUs, reported speedups range from 1.30× to 1.68× on H20 and from 1.76× to 2.17× on H100. Separate experiments on 16 L20 GPUs, using 350-million- and 1.3-billion-parameter models, report speedups between 1.72× and 2.08×.

Where the gains are constrained

Independent streams can compete for shared PCIe links. RM+ alone performs worse than Ring-Attention in the reported L20 ablation configurations. The complete CHARM+ approach improves performance in the tested cross-NUMA settings, but concurrent communication is not automatically beneficial.

Workload balance matters too. In the 16-H100 experiments with sequences exceeding 200,000 tokens, computation takes longer than communication. The authors report smaller speedups there because hiding communication addresses less of the total runtime. Non-attention layers also limit how much faster attention can accelerate full training.

The method requires extra buffers, and measured memory consumption is somewhat higher in the reported comparisons. Numerical checks against FlashAttention, using 100 randomly generated input sets on eight L20 GPUs, find output and gradient differences of the same order as Ring-Attention’s.

The results concern execution efficiency, not improvements in language-model quality. Their practical message is that communication scheduling can substantially affect distributed attention performance, but the benefit depends on the links connecting the GPUs and the balance between calculation and data transfer.