A smarter way to shrink video memories

Teaching a computer to mark where one action ends and another begins in a long video requires following events over time. A system might need to distinguish taking an ingredient from pouring it, then identify where each action occurs. This task, called temporal action segmentation, is often trained using a feature vector for every video frame. These precomputed features make training practical, but storing them can become burdensome, especially when a model learns new tasks over time and must replay examples from earlier ones.

Gerard Donahue and colleagues propose a way to compress this feature memory without using action labels to choose how many codes each frame receives. Their method, Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE), is tested on action-segmentation benchmarks including Breakfast, 50Salads and GTEA.

A variable-length code for each frame

The method converts each frame’s feature vector into a sequence of discrete codes. The first code gives a rough representation; later codes represent residual information left over. This approach is called residual quantization.

A standard version stores the same number of codes for every frame. ARQ-VAE instead evaluates successive prefixes of each frame’s code sequence and keeps the prefix that gives the lowest reconstruction error. A short prefix may reconstruct a frame well; another frame may have lower error with more codes. This selection rule does not directly minimize storage or guarantee the shortest adequate representation. The authors then fine-tune the decoder and codebooks—the learned collection of possible codes—to improve reconstruction using the prefixes that will actually be stored.

The selection is label-free: it relies on reconstruction error, not action labels or manually marked boundaries. During later training, the codes are decoded into approximate features. The segmentation model itself is unchanged: it trains on reconstructed features and is evaluated on original test features.

Storage falls sharply in benchmark tests

On the Breakfast benchmark, the authors report reducing the original I3D feature bank from 28 GB to 10 MB. With the ASFormer segmentation model, the fine-tuned representation achieved an Edit score of 71.5 and an F1@50 score of 51.2 in the supervised setting. These metrics assess aspects of predicted action segments, and results vary across datasets, models and measures.

The authors also test incremental segmentation, where a model learns tasks in sequence, and unsupervised segmentation, which does not use frame-level action labels during downstream training. They report that the compact memory remained useful in both settings. In incremental experiments, outcomes depend on the task setting, backbone and metric; the tables show mixed comparisons with prior replay methods rather than a uniform advantage. Unsupervised results also vary by dataset and metric: on GTEA, some compressed-feature results matched or exceeded the original-feature reference, while other comparisons were closer or lower.

One analysis asks whether the placement of codes matters. On a single 50Salads split, assigning code depths according to reconstruction error performed better than randomly shuffling those same depths among frames. This suggests that frame-specific allocation matters, not just the total number of codes. The paper also reports that frames with greater temporal redundancy tended to receive shorter prefixes. These are observed associations, not a guarantee that every action transition or visually complex frame gets a longer code. Nor do the codes identify actions: the authors note that codes do not correspond one-to-one with action labels or segments.

What the results leave open

A key accounting detail is that the main storage figures exclude the decoder needed to reconstruct features. In a separate 50Salads comparison that includes the decoder, the authors report 5.521 MB for ARQ-VAE. Storage also depends on the dataset and experimental setup, so the headline comparison is not a complete accounting of every component in every deployment.

ARQ-VAE compresses features already extracted by a video backbone; it cannot recover information that the backbone did not capture. And reconstruction error is a proxy for what segmentation needs, not a direct measure of segmentation quality. The authors identify task-specific code selection as a possible future direction. Tests across several benchmarks and architectures do not establish that the method will work equally well for other video tasks or datasets.

The source is an arXiv preprint, version 1. The paper text identifies NeurIPS 2026, but the available evidence does not independently establish its peer-review status or confirm later revisions. The results should therefore be understood as the authors’ reported experiments, not settled evidence. They suggest that adaptive feature storage could ease a practical memory bottleneck in temporal action segmentation, while broader generalization and complete deployment costs remain open questions.