The timing trick that lets LLM lookup tables move to disk

A memory problem with a timing solution

Some language models use large lookup tables alongside their neural-network weights. DeepSeek-V4.1’s two Engram tables contain information indexed by short sequences of input tokens. Together, the tables occupy about 203 GB. In the authors’ four-GPU serving setup, keeping them in host memory accounted for 190 GiB of a process using 223 GiB in total.

The challenge is not just where the tables fit. It is when their data must arrive. A lookup’s row indices can be calculated from the input tokens before the model’s forward pass begins. That lets a server start reading rows while earlier layers are computing. But different layers provide different amounts of time for those reads to finish.

The authors measured a striking gap in DeepSeek-V4.1: the first Engram layer has a prefetch window of roughly 0.3 milliseconds, while the later one has about 9.3 milliseconds. Think of these windows as delivery deadlines. A slow storage device may meet the later layer’s deadline but miss the earlier one, even when both tables are similarly large.

Scheduling reads, not just caching popular rows

The researchers built SlackTier, a system for the SGLang serving framework that can place lookup tables in host memory, local NVMe solid-state drives, or another machine’s memory accessed over RDMA (remote direct memory access). It starts lookups at the beginning of each decoding step, prioritizes the early layer’s reads, and divides a small GPU cache between the layers.

This design responds to two complications. First, requests within a conversation often reuse some lookup rows, making a cache useful. But rows that are popular across sessions are not necessarily the ones a particular session will need: the authors found that a dynamic least-recently-used cache captured more of the tested traffic than fixed lists of supposedly “hot” rows. Second, the two layers’ reads can compete for the same disk. A delay to the early layer can also give the later layer more time, so simply adding each layer’s delay separately can misstate the total slowdown.

To account for this, the paper presents a model that combines measured read times, each layer’s deadline and interference between reads. The authors tested the system on DeepSeek-V4.1 and a second model, Qwen3.8, using agent and chat request replays. They also checked that SlackTier returned the same lookup values as the original gather operation: their in-server checks reported no mismatches across the tested lookups, including runs with cache evictions.

What the tests found

For DeepSeek-V4.1, the later layer’s table could be read from NVMe without a cache and without measurable decode slowdown in the reported tests. The early layer was more sensitive. Splitting a small GPU row cache across both layers worked substantially better than putting it only on the later layer, because the early layer has less time to wait for missed rows.

With both tables on four NVMe drives, the authors report that mean time per output token rose by at most 1.4% on their agent and chat replays compared with unmodified SGLang. Host-memory use fell from 212 GiB in that comparison to 34 GiB. Reading both tables from another node’s memory over RDMA required no cache at higher tested session counts and had performance close to the host-memory baseline. These outcomes depend on the storage setup and the workload; they are not a promise that any disk or network link will behave the same way.

The reclaimed memory also mattered in one constrained agent-serving experiment. When the team used it for host-side key-value (KV) cache—the stored context used to continue conversations—median throughput increased by 36%, and median 90th-percentile time to first token fell from about 38 to 24 seconds. This is a result of that particular memory-budget experiment, not evidence that moving tables to disk alone makes a model faster.

What remains uncertain

The main lesson is a systems principle: choose storage according to when each layer needs its data, not only according to table size or average row popularity. But the evaluation has limits. End-to-end tests focused on two models and four H200 GPUs, with specific SGLang software, storage devices and replayed traffic. Some measurements on other hardware were probes of timing windows rather than full end-to-end evaluations. The authors also note that new models and deployments need their own measurements of lookup deadlines, read speeds, cache behavior and contention.

The paper is an arXiv preprint, not a peer-reviewed publication. Its figures describe experiments reported by the authors; independent replication would help establish how well the approach transfers to other serving systems and workloads. SlackTier targets sparse, token-indexed lookup tables whose addresses are known in advance. The authors explicitly distinguish these from dense model weights, which require much broader data transfers and cannot be assumed to benefit from the same approach.