A Better Timetable for Training Reasoning AI

Why fast chips still spend time waiting

In reinforcement-learning post-training, a language model generates responses and receives signals that guide its next update. But the work involves more than the model answering questions.

In a common method called Proximal Policy Optimization, or PPO, an actor generates responses, a reward model scores them, a critic estimates future returns, and a frozen reference model helps constrain changes in behavior. Training then updates the actor and critic.

These jobs have different appetites for computing resources. Generating text proceeds token by token; assessing completed responses allows more parallel work; updating models requires extra calculations and memory. Making each job individually faster does not necessarily produce the fastest overall training cycle.

That coordination problem is the target of RPEG, the RLM Pipeline Execution Graph. Maciej Besta, Leonard Schmidt, Pierre Pang, Patrik Okanovic, Robert Gerstenberger and Torsten Hoefler, at ETH Zurich, describe it in a preprint posted to arXiv on November 6, 2026. Its findings should be treated as author-reported preprint results, not established independent validation.

Turning training into a searchable map

RPEG represents the pipeline as a graph: operations are connected by dependencies showing which data and model updates they need. Those connections include dependencies between successive training cycles, not just jobs within one cycle.

The system searches both how to organize selected parts of the pipeline and how to execute them. It can choose which graphics processors, or GPUs, run each operation, how to divide models across them, and how to split batches into smaller pieces.

One useful opportunity comes from responses finishing at different times. Rather than waiting for the longest answer before assessing anything, RPEG can arrange for completed responses to be evaluated on separate GPUs while longer responses continue generating.

Other choices involve deeper tradeoffs. Bounded asynchrony allows generation or assessment to use a specified older model version while training advances. Actor–critic sharing replaces separate model backbones with a shared one and distinct output heads. Sharing changes the model architecture; asynchrony changes parameter freshness. Neither is simply an interchangeable scheduling trick.

Predict first, execute on completion

Testing every possible arrangement on a supercomputer would be expensive. RPEG instead estimates calculation time, communication costs and memory requirements, using measurements from the target hardware rather than advertised peak specifications.

The estimates account for where GPUs sit: moving data within a machine differs from sending it between machines or racks. Memory accounting also considers multiple models residing on the same GPU.

A stochastic search explores candidate plans, sometimes accepting a temporarily worse choice to escape an unproductive arrangement. The selected final plan must satisfy the model’s memory constraints. During execution, actual completion events—not predicted timestamps—determine when dependent work can begin. Runtime traces check placements, dependencies and required parameter versions.

What the experiments show

The authors report physical execution on systems with up to 256 NVIDIA GH200 GPUs. In synchronous PPO comparisons on A100 hardware, RPEG delivered up to twice the throughput of ReaL. Against verl on GH200 hardware, with separate actor and critic models, speedups ranged from 1.024 to 2.477 times, with a median of 1.488 times. These are workload-specific comparisons, not universal multipliers.

A useful additional test kept the runtime, architecture, synchronous execution and GPU budget fixed. Across 20 measured comparisons using Llama 3.1 8B, optimized plans were 1.250–1.471 times faster than their fully colocated baselines. That helps separate planning improvements from differences between software frameworks.

The prediction model was checked against 123 configurations. Mean absolute percentage errors were 15.5% for operation latency and 15.1% for peak memory. Its strong ranking performance matters because selecting a better plan does not require predicting every duration exactly.

The boundaries of the claim

Some larger-scale results are predictions: optimizer studies extending to 2,048 GPUs are not demonstrations of training at that scale. Scheduled GPU occupancy also differs from measured arithmetic utilization, and neither directly establishes electricity or financial savings.

For training outcomes, asynchronous RPEG achieved a final mean rollout reward within 0.47% relative difference of synchronous verl under the evaluated settings. That is narrower evidence than proving equivalent reasoning ability across tasks or long training runs.

The paper also acknowledges approximate communication and memory models, hardware-specific calibration and overheads that estimates do not fully capture. The supplied HTML extraction omits figures and can distort tables and notation.

The practical proposition remains clear: training efficiency depends on the whole workflow. RPEG offers evidence that coordinating computation, communication and waiting together can outperform optimizing each in isolation.