MeshRT keeps dynamic AI inference control on wafer-scale chips

Why fast chips need different software

Wafer-scale accelerators bring vast numbers of processing cores and fast local memory onto one enormous chip. That arrangement can reduce delays from fetching model data or communicating between separate devices. But it also creates a coordination problem: the cores have limited local resources and no shared memory through which to manage changing work.

In an arXiv preprint, researchers from the University of Edinburgh and Microsoft Research present MeshRT, a system designed to handle those changes without repeatedly asking a host computer what to do. Their central idea is to compile rules for dynamic execution into each core’s schedule beforehand. The supplied manuscript also lists an SOSP 2026 proceedings reference.

The goal is not simply faster generation for one user. It is higher aggregate throughput—the total tokens generated across concurrent requests—while retaining low per-token latency.

Preparing for decisions that cannot be predicted

Two important serving techniques create unpredictable work. Continuous batching replaces completed requests with waiting ones instead of leaving empty slots. Mixture-of-experts models route each token to selected neural-network components, or experts, whose identities become known only during execution.

MeshRT does not predict those outcomes. Instead, it uses the known computation graph and placement of operations to determine where decisions arise, which cores need them and how those cores should respond.

The compiler inserts small control-handling routines around ordinary computation. Compact messages carry information such as request status or expert destinations alongside model activations—the intermediate numerical representations passed between operations. Cores then execute their own schedules asynchronously, coordinating through arriving data rather than a central per-token scheduler.

The host still has a role: it installs the schedules and asynchronously supplies requests and their data. MeshRT moves per-step scheduling and dynamic control onto the accelerator; it does not eliminate the host altogether.

Making the schedules fit safely

The implementation runs on the Cerebras CS-3, powered by a WSE-3 processor with approximately 900,000 cores and 44 GB of aggregate on-chip SRAM. Each core has only about 48 KB of local memory, along with limited routing and queue resources.

A full schedule can demand more active hardware state than a core can accommodate. MeshRT therefore divides schedules into segments and activates their routing and queue configurations incrementally. The compiled program image remains resident; segmentation does not remove its memory footprint.

Switching configurations requires care because data may still be travelling through neighboring routers. MeshRT uses suitable collective-operation boundaries as safe transition points. Elsewhere, it inserts bounded synchronization within the affected region. This is intended to prevent configuration changes from corrupting in-flight traffic and causing deadlock without imposing chip-wide barriers.

What the reported gains mean

The evaluation covers Qwen3.5-9B, Qwen3.5-35B, GPT-OSS-20B and GPT-OSS-120B, comparing MeshRT with the wafer-scale serving system WaferLLM and GPU serving through SGLang on NVIDIA H200 hardware.

At each system’s highest-throughput decode operating point in the main sweep, MeshRT’s reported gains over WaferLLM range from 6.3 to 55.8 times. These results reflect enabled policies, including overlapping micro-batches through a pipeline and parallel expert execution, rather than a speedup to every individual operation. The plotted comparisons exclude additional workload-level padding savings from continuous batching.

Dynamic control also has a cost. In matched configurations for the 9B and 20B models, MeshRT adds approximately 40 and 25 microseconds per generated token, respectively—latency increases of 9.5% and 7.6% over WaferLLM.

Crucially, the results combine measurements with calculations. The authors compose full-model throughput from measured device-component cycles for the 9B and 20B configurations. For the 35B and 120B models, they project larger placements from component measurements on two wafers.

Against measured single-instance GPU serving, latency-oriented MeshRT configurations deliver 4.5–7.0 times the per-user decode token rate. The reported 1.7–13.0-fold aggregate-throughput advantage instead compares against an ideal linear scale-out of GPU measurements to an approximately power-matched cluster—not a measured physical cluster. Energy-efficiency estimates use vendor power budgets, not measured workload electricity consumption.

Where the advantage stops

MeshRT is not uniformly better. For throughput-oriented decode, its projected aggregate throughput on Qwen3.5-35B is only 0.58 times the GPU projection. GPUs also retain a substantial aggregate-throughput advantage in prefill, the initial processing of prompts, where matrix-computation capacity matters more.

The design assumes exclusive mesh access for one model and fixed, preloaded weights. It does not recover from runtime core failures, and unfamiliar operator types may require new implementations. Globally coupled decisions still incur communication costs and can stall dependent work.

The contribution is therefore specific: a way to support changing inference workloads within tightly constrained, precompiled wafer-scale execution—not proof of universal GPU replacement or measured serving-cost savings.