AMD researchers accelerate training workloads by keeping GPU data moving

A powerful GPU can still lose time waiting for data—or spend computing resources moving it. AMD researchers investigate how dedicated transfer engines and specialized loading instructions can reduce those costs, with experiments that separate individual kernel improvements from gains across training workloads.

Training a large language model requires more than fast arithmetic. Data must arrive from memory, take the layout needed by each calculation, and move between GPUs participating in the same job. In the preprint Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs, Gengyu Rao and colleagues at AMD investigate how hardware-assisted data movement can keep those calculations supplied.

The paper, submitted to arXiv on November 3, 2026, describes implementations in AMD’s open-source Primus training framework and Primus-Turbo acceleration library. Its experiments cover two distinct settings: attention kernels on an MI455 bring-up platform, and communication–computation overlap on MI355 GPUs. Their results should not be treated as one combined training acceleration.

Moving data without occupying the compute units

Within a GPU, the researchers use the Tensor Data Mover, or TDM, to transfer tensor data between high-bandwidth memory and a small, fast on-chip scratchpad called Local Data Share. Rather than making computing threads execute the individual loads, software schedules transfers through dedicated hardware and waits for the required data to become ready.

This is especially useful for the paper’s case study: Compressed Sparse Attention configurations from the DeepSeek V4 architecture. These calculations gather selected key and value rows rather than simply reading one continuous block. TDM supports indexed gathering, allowing the kernels to prepare upcoming data while calculating with earlier tiles.

The comparison covers Flash and Pro configurations, three compression settings, and sequence lengths of 4K, 8K and 16K tokens. Against the authors’ best non-TDM implementations on the same MI455X, TDM produces a geometric-mean speedup of 1.88× for the attention forward-plus-backward step, reaching 2.74× in the strongest configuration. These are attention-kernel measurements, not whole-model training results.

The authors attribute the improvement partly to register availability. Their non-TDM design uses two waves of threads per SIMD execution unit to hide memory latency. With TDM, the kernels use one wave and can devote more registers to each wave, supporting deeper pipelines and greater data reuse.

Changing layout while loading

A second optimization addresses matrix transposition. Training’s forward and backward calculations need different operand layouts. Rearranging values through register shuffles or loading them with a strided access pattern adds overhead.

Specialized transpose-load instructions instead rearrange data as it moves from on-chip scratchpad memory into registers. In a bandwidth comparison normalized to a plain load, the transpose-load implementation reaches 0.99, versus 0.63 for strided loads and 0.41 for loading followed by register shuffles.

For the attention-kernel comparison, the researchers retain TDM and replace only the transpose-load path with strided loads. Excluding configurations where the replacement causes register spilling, they report a geometric-mean step speedup of 1.19×, with a maximum of 1.21×. Some table entries show larger improvements, but those include spill-related penalties in the comparison implementation and should not be conflated with that headline result.

Communication still has a cost

Between GPUs, the work examines System DMA for local scale-up transfers and Remote DMA for transfers through network interfaces across nodes. The objective is to avoid assigning GPU compute units to data movement while useful calculations are running.

In MI355 matrix-multiplication microbenchmarks, conventional compute-unit-based communication increases runtime by reported geometric means of 28% during AllGather and 35% during ReduceScatter. With SDMA, those increases fall to 8% and 14%. The overhead does not disappear: the SDMA ReduceScatter implementation still uses a local kernel for reduction.

Whole-training gains are smaller. Across eight GPUs, SDMA raises throughput by 2.1% for Llama 3.1 8B, 4.5% for GPT-OSS 20B and 3.1% for the FLUX.1 diffusion workload; FLUX enables SDMA only for AllGather. The paper’s approximately 1.16× figure concerns the overlapping communication-and-computation regions, inferred using the iteration breakdown and Amdahl’s law—not entire training runs.

What the evidence establishes

The multi-node results are preliminary. Hierarchical SDMA and RDMA integration into training workloads remains underway, so the scale-out evaluation uses matrix-multiplication overlap microbenchmarks on 8, 16 and 32 MI355 GPUs. On 16 and 32 GPUs, overlapping matrix multiplication is about 1.33× faster with hierarchical DMA than with compute-unit-based collectives.

The study also reports only relative performance, citing company policy. Its MI455 measurements come from a bring-up platform and are explicitly described as a point-in-time snapshot. Together, the experiments support targeted benefits from better data scheduling, while leaving full-training scale-out benefits unestablished in this evaluation.