Learning to reconstruct single-pixel video from noisy-looking feedback

Single-pixel imaging replaces an array of camera pixels with a sequence of known spatial patterns and one detector. For each pattern, the detector records a total-light measurement called a bucket value. Reconstructing a scene means working backward from those numbers. In their November 1, 2026 preprint, Hongjun An, Shuaizhe An, Tong Tian and Zhe Sun investigate how to make that reconstruction faster for short video clips.

Their target setting is demanding: five grayscale frames at 856 × 480 pixels, with 1,000 bucket values per frame. That measurement count is about 0.24% of the number of pixels, not a demonstrated video-file compression ratio. Simply correlating the measurements with their patterns produces a noise-like map rather than a usable scene image.

Turning an estimate’s mistakes into feedback

The authors instead begin with an estimated video. Because the sensing patterns are known, they can calculate which bucket values that estimate should produce. Subtracting the acquired values gives a measurement residual. Correlating the mean-centered residual with the patterns creates what they call a residual image.

This image has a precise mathematical role: it is the gradient of a centered squared measurement-error objective. It indicates how changing the estimated pixels would change that error. Yet it also looks noisy, and the sparse measurements leave most possible pixel changes unconstrained.

The researchers pass this feedback backward through a pretrained video decoder into its compact internal representation, or latent space. In their diagnostic experiments, the corresponding change in decoded video agrees more closely with the true reconstruction error than the original residual image does. Reinitialized and weight-shuffled decoders perform worse, supporting the authors’ interpretation that learned video structure matters. This transformation adds no new measurements and does not guarantee recovery of unseen details.

Training once rather than optimizing every clip

The team first develops video-latent inversion, or VLI, which optimizes a reconstruction network and decoder separately for each clip. Jointly reconstructing frames improves low-measurement performance relative to frame-by-frame image-latent reconstruction, but remains slow.

Their faster method, Measurement-Gradient-Conditioned Video Flow, or MGV-Flow, is trained once. It starts from random noise and progressively updates a video latent using the encoded preceding five frames, measurement-derived feedback and descriptors of that feedback. Rather than adding a fixed correction, the model learns how to incorporate it.

In a matched benchmark of 21 cases on an RTX 4090, MGV-Flow reconstructed five frames in a median 1.55 seconds using five integration steps. VLI took 397.99 seconds, giving a median case-wise speedup of about 256 times. Median peak signal-to-noise ratio, an image-error measure where higher values indicate closer agreement with reference frames, rose from 18.12 to 22.92 decibels. These timings started with inputs in memory and excluded acquisition, disk operations and model loading.

VLI nevertheless matched the bucket measurements more closely. The result illustrates the ambiguity of sparse sensing: reducing measurement error alone need not identify the correct video.

What the measurements actually contribute

The large improvement over VLI should not be confused with an equally large benefit from measurement feedback. Across a separate 105-case evaluation, MGV-Flow and a separately trained history-only model were within 0.15 decibels on average. At five steps, the history-only model was slightly better.

Measurement feedback was more useful as motion increased and preceding frames became less predictive. Replacing the correct measurements with another video’s measurements reduced MGV-Flow’s average quality, demonstrating sensitivity to the recorded scene. However, on an unseen dataset, an explicit gradient-correction baseline outperformed it when allowed ten steps. The learned method was not universally best.

The authors also tested an offline pattern-selection measure called directional fidelity. Across 30 coding configurations, it had a Spearman rank correlation of 0.779 with quality after ten measurement-driven latent updates. A selected configuration matched or exceeded a median-ranked comparator using 2,000 rather than 3,000 measurements. This analysis required ground truth and a ground-truth-derived starting latent; whether the selected patterns improve MGV-Flow itself remains untested.

Laboratory evidence and remaining limits

A physical experiment used two ten-frame animation excerpts acquired frame by frame. Without retraining, MGV-Flow outperformed the tested per-frame reconstruction methods. After independent histogram equalization, its mean PSNR was 10.25 decibels, compared with 8.63 for the strongest per-frame comparator. Accurate reference history supplied most of its performance; swapping measurements reduced quality by just 0.16 decibels on average.

The simulations were noiseless, training used windows from 17 source videos, and the reported reconstruction evaluations relied on ground-truth history frames. Continuous reconstruction using its own previous outputs therefore remains unestablished. The measured latency corresponds to about 3.2 output frames per second; estimates above 20 on a data-centre accelerator are theoretical. Reconstructing history and addressing detector calibration remain necessary for a complete end-to-end system.