Omni-Encoder tests one shared backbone for images, video and sound

Omni-Encoder aims to replace several separate perception backbones with one shared network. In an arXiv preprint, Xiaofei Sun and colleagues at Alibaba Group report results across image, video, audio and combined audio-video tasks. Their evidence supports a qualified conclusion: one jointly trained checkpoint can remain competitive across these inputs, but it does not lead on every benchmark or preserve every capability of its image-model starting point.

The system is a representation encoder, evaluated on classification and retrieval rather than conversational response generation. It learns numerical representations that can be compared with representations of text. For retrieval, the goal is to rank a matching item highly among candidates; the paper reports recall at one, or R@1, the percentage of queries whose correct match appears first.

Shared processing, different entrances

The central network is a SigLIP-2 SO400M Vision Transformer initialized from pretrained image-text weights. Images, videos, audio and audio-video sequences pass through the same Transformer parameters. Text is handled by a separate text tower, so the sharing claim concerns the four perception input groups, not a single network that also processes language identically.

Sharing begins after input preparation. Images and videos use a visual tokenizer that processes pairs of frames; a still image is duplicated to fit that arrangement. Audio takes a different route: recordings become mel-spectrograms, which describe sound energy across frequency and time, and a one-dimensional convolutional tokenizer converts them into tokens. Its filters operate along time while retaining the full frequency spectrum at each step, rather than cutting the spectrogram into image-like patches.

The resulting tokens enter a common embedding space. This distinction matters: Omni-Encoder shares its main semantic processing network without forcing fundamentally different signals through an identical front end.

Putting sound and vision on one clock

A second design choice concerns timing. The model extends rotary position embeddings to include time alongside spatial coordinates. Temporal positions use timestamps in seconds rather than frame numbers. A visual observation and an audio segment occurring at the same time can therefore receive corresponding temporal coordinates despite different sampling rates.

For combined audio-video inputs, the model alternates groups of video and audio tokens within one sequence. This places corresponding segments next to each other and allows attention between them inside the shared encoder. The authors argue that this arrangement makes temporal correspondence easier to learn. Their broader claims about generalizing across sampling rates and longer durations should be distinguished from the benchmark scores reported in the results table.

Training also alternates modalities. Each optimizer update follows separate image, video, audio and audio-video forward-and-backward passes, accumulating their gradients before changing the weights. The sample ratio is 64:16:64:16, not equal numbers of examples. The authors describe this schedule as a way to reduce competition between modalities while accommodating their different input shapes and processing costs.

Where the results hold up—and where they do not

The paper reports 16 benchmark evaluations using a single checkpoint in a zero-shot setting. On MSR-VTT video-text retrieval, Omni-Encoder reaches 54.20% R@1, compared with 51.40% for VideoPrism-B in the comparison table. On ActivityNet, the corresponding scores are 53.50% and 49.60%. Larger embedding systems nevertheless lead on several video tasks.

AudioCaps retrieval reaches 42.20% R@1, narrowly above GLAP’s 41.70%; both score 19.40% on Clotho. An important protocol detail is that Omni-Encoder’s audio retrieval uses an additional training-free score-calibration step at inference. Other tasks use raw cosine similarity scores, so the audio figures should not be described as results from an entirely unchanged scoring procedure across modalities.

Image results reveal trade-offs. ImageNet-A accuracy rises to 84.12%, from 79.32% for the listed SigLIP-SO400M baseline. ImageNet-1K falls from 84.10% to 83.08%, however, and ImageNet-V2 and Sketch also decline. These results do not support a blanket claim that joint training improves every task.

What the evidence establishes

The authors interpret the overall pattern as positive cross-modal transfer. However, the reported comparisons do not isolate every architectural and training contribution through controlled ablations, so the results establish the combined recipe’s performance more directly than the independent benefit of each component.

The paper also acknowledges serving costs: a deployment using only one modality still carries the full shared backbone. It identifies remaining gaps on specialized audio tasks and proposes distillation, pruning and additional data as future directions. Code and models are promised for public release, not established as available by the supplied source. The work is presented as a preprint, with its findings attributable to the authors rather than treated as independently validated conclusions.