VideoMogaNet targets cheaper motion modeling for video recognition

A video-recognition system must connect appearance with movement. A frame can reveal a person’s limbs or an object’s shape, but recognizing an action can depend on how those features change across several frames. VideoMogaNet proposes a relatively inexpensive way to make those connections.

Kevin D. Nguetche, Lilian Hollard, Eric Granger and Marco Pedersoli describe the architecture in Spatio-Temporal Multi-Order Interaction for Video Recognition, an arXiv preprint submitted on November 3, 2026. Their approach extends MogaNet, an image-recognition architecture, with a lightweight temporal module and a different way of combining spatial and temporal information.

Connecting appearance and motion

The authors’ motivation centers on “multi-order” interactions. In the paper’s game-theoretic framework, interaction order concerns the amount of surrounding context involved in an interaction between visual patches. The authors associate middle-order interactions with structured compositions, such as coordinated body parts, rather than isolated textures or whole-scene context.

VideoMogaNet aims to strengthen these intermediate relationships without discarding local or global information. Its spatial component builds on MogaNet’s existing multi-order processing. The new Multi-Order Temporal Modulation module, or MOTM, handles changes across frames.

At each spatial position, MOTM treats the features across frames as a temporal sequence. It divides feature channels into three groups in a 1:3:4 ratio and processes them in parallel with depthwise convolutions using different kernel sizes and dilations. These filters provide different temporal receptive fields. Depthwise processing filters each channel independently, avoiding explicit cross-channel projections within the temporal module.

Two gating branches then combine the normalized sequence with the extracted temporal features through element-wise multiplication. The resulting output is a modulation signal, not a separate action prediction.

The second contribution, Spatio-Temporal Multi-Order Interaction, or ST-MOI, uses that signal to adjust spatially and channel-enriched features. Instead of simply adding temporal features, it applies a multiplicative adjustment controlled by a learnable scale and a bounded activation. The scale starts at zero, so temporal modulation is introduced progressively during training. Temporal processing also comes after channel mixing, allowing it to operate on features already enriched with inter-channel information.

Results across four benchmarks

The paper evaluates VideoMogaNet on UCF101, HMDB51, Kinetics-400 and Diving48. Its Base model reports 97.1% top-1 accuracy on UCF101 and 74.7% on HMDB51, the highest scores among the methods included in that comparison. Those evaluations use eight-frame clips, with predictions averaged over four temporal clips and three spatial crops.

On Kinetics-400, Base reaches 80.6% and Large reaches 81.3%, also using eight-frame clips. These results exceed several listed convolutional, transformer and state-space baselines, but they are not the highest in the table. Larger models, including VideoMAE variants and InternVideo, report higher accuracy. The authors characterize VideoMogaNet’s result here as a competitive accuracy–cost trade-off rather than an overall accuracy record.

Diving48 provides a particularly relevant test because its classes depend on subtle differences in diving movements against relatively consistent backgrounds. With 16 frames and single-view evaluation, Base reaches 90.30%, compared with 88.9% for PosMLP-Video-L in the paper’s comparison. A separate temporal-scaling experiment reports 84.64%, 90.30% and 93.60% with eight, 16 and 32 frames respectively. Competitive short-clip performance therefore does not mean additional frames are unnecessary.

Ablations help distinguish the design choices. On Diving48, multiplicative ST-MOI fusion improves accuracy from 89.5% with additive fusion to 90.3%. The parallel temporal design also outperforms the tested serial alternatives.

What the efficiency claim covers

The temporal module’s cost grows linearly with channel count, but the full network retains quadratic channel-mixing costs in its spatial backbone. In the reported Kinetics-400 temporal-module comparison, MOTM achieves 80.6% accuracy versus 80.9% for a direct temporal adaptation of MOGA, at approximately 70 times lower temporal computation. That is not a 70-fold reduction in total inference cost.

Evaluation protocols also matter: Base’s reported 0.953 TFLOPs on Kinetics-400 includes 12 views, while the supplementary architecture table lists 79.398 GFLOPs per view.

The interaction analysis offers qualified support for the proposed mechanism. VideoMogaNet modestly strengthens middle-order interactions relative to MogaNet with temporal averaging, but the overall profiles remain similar and still favor extreme orders. The main text explicitly describes the changes as gradual, not qualitatively distinct.

Benefits also vary by action class, with both gains and losses reported when adding MOTM. Finally, the authors limit the demonstrated scope to single-person, clip-level action recognition. They identify multi-person recognition as requiring additional architectural adaptations, rather than an already established capability.