AdaOffload adapts CPU and GPU workloads to run large AI models

Running a large AI model involves more than providing enough computing power: its learned weights must also fit somewhere. Mixture-of-experts, or MoE, models divide part of their computation among networks called experts. Each token—a unit of text processed by the model—activates only a few experts, but the full collection can still require hundreds of gigabytes of storage.

AdaOffload addresses a practical question: when expert weights cannot all remain in GPU memory, how should a server share work between its CPUs and multiple GPUs? William X. Zheng and colleagues describe the system in an arXiv preprint dated November 6, 2026. The supplied source does not establish peer-reviewed publication.

Why moving less can help

Expert offloading keeps some weights in host memory, the system memory accessible to CPUs, while retaining a subset on GPUs. When an absent expert is needed, a server can fetch its weights for GPU execution. Alternatively, it can run that expert on a CPU and transfer the smaller intermediate inputs and outputs instead.

Neither option always wins. Fetching weights consumes bandwidth on PCIe, the connection used for CPU–GPU transfers. CPU execution can become a bottleneck, and both activities compete for host-memory bandwidth. Adding GPUs therefore does not automatically produce a proportional speed increase.

The trade-off also changes between inference phases. Prefill processes the input prompt; decode generates subsequent tokens. Longer prefills may provide enough computation to hide transfers of the next layer’s expert weights. Shorter prefills and decode may benefit from sharing expert computation between CPUs and GPUs.

How AdaOffload chooses

The system combines hardware profiling with observations collected during inference. Before serving, it measures expert computation and fetching under concurrent activity, rather than assuming that timings measured separately will hold when resources compete.

During operation, sampled passes record which experts are used, how many tokens reach them and which weights are already cached on GPUs. A background selector estimates the cost of different policies, choosing what fraction of missing experts to fetch and what fraction to execute on CPUs. For prefill, it also considers fetching all experts for the next layer in advance.

AdaOffload distributes pieces of each expert across GPUs using tensor parallelism. It partitions CPU work independently, accounting for non-uniform memory access, or NUMA: a hardware arrangement in which a CPU’s access speed depends on where memory resides. This separation allows CPU and GPU partitions to suit their different hardware constraints.

What the tests found

The evaluation covers three open-weight models with 284 billion to 753 billion parameters and five platform configurations spanning L4, PRO 6000 and B200 GPUs. Workloads include AIME mathematics problems and multi-turn coding tasks from SWE-bench Verified, with one, four or eight concurrent requests.

For DeepSeek-V4-Flash, comparisons used KTransformers, llama.cpp and Ollama, with matched model checkpoints, quantization and expert-cache capacity. Quantization stores weights at reduced numerical precision. KTransformers was omitted from the GLM comparisons because it lacked the relevant native model or format support.

On DeepSeek-V4-Flash mathematics workloads, average decode-throughput improvements across four platforms were 1.30, 1.88 and 2.47 times the strongest evaluated baseline at concurrency levels of one, four and eight, respectively. Mean time to first token—the wait before a response begins—fell by 41%, 57% and 67%.

A separate scaling experiment used eight concurrent mathematics requests on a GCP machine. AdaOffload’s reported decode throughput increased from 28.5 tokens per second with one L4 GPU to 90.8 with eight. Its adaptive policy reached 94.8% and 99.2% of the best measured fixed-policy throughput at concurrency one and eight, averaged across four platforms. Those comparisons concern tested policies, not a proven global optimum.

Where the evidence stops

These are serving-performance results, not evidence of improved reasoning accuracy or coding success. The gains depend on hardware, model, concurrency and baseline support. Coding comparisons also involved frequent baseline session timeouts and prefix-cache misses, so they do not isolate expert scheduling alone.

Adaptation has costs: instrumented passes incurred an average 4.04% extra latency, while background decode-policy selection averaged 4.1 milliseconds. The preprint’s headline speedup range also differs from individual results and averages in its evaluation; specific comparisons are therefore more informative than a universal multiplier. Overall, the evidence supports adaptive CPU–GPU coordination for the tested deployments, not guaranteed gains on every server.