Learning Video Timing From Text Reveals a Persistent Modality Gap

A model may understand descriptions of an unfolding scene without knowing how to interpret the scene’s visual features. That distinction drives Grounding in the Dark: Text-Only Training for Zero-Shot Video Temporal Localization, by Divyanshu Mishra, Sabine Sternig, Rakshith Shetty and Erhan Gundogdu. The work is an arXiv preprint; the supplied source reports no peer-reviewed acceptance.

The task is temporal grounding: given a video and a query about an event, predict its beginning and end. The researchers ask whether a model can learn this task from text descriptions and encounter actual video only at inference. Their answer is qualified: transfer improves when visual features are mapped back into text representations, but substantial localization errors remain.

Teaching time without training on video

The team starts with Qwen3-8B, a text-only language checkpoint rather than an existing video-language model. They build Temporal-TextVid, a synthetic corpus containing 134,000 examples, by using Qwen3.8-27B to expand source descriptions into sequences of up to 100 textual time steps. Generated questions cover localization, descriptions, event ordering and multiple-choice tasks.

Each time-step description becomes an embedding—a numerical representation—through a frozen CLIP or LLM2CLIP text encoder. A learned projection converts that representation into the language model’s input space, while learned temporal embeddings mark sequence positions. Training updates these components and low-rank adaptation, or LoRA, parameters; the original language-model weights remain frozen.

At inference, the corresponding vision encoder processes sampled video frames. The downstream model stays unchanged. Importantly, “text-only” describes the temporal model’s training, not the entire system’s history: the frozen encoders already provide image–text alignment learned through multimodal pretraining.

Good text performance does not guarantee visual transfer

Detailed descriptions worked better with LLM2CLIP than CLIP. On the synthetic text evaluation, the detailed-description models scored 61.54 and 53.86 respectively in mean temporal intersection over union, or mtIoU. This metric measures overlap between predicted and reference intervals, averaged across examples and reported on a 0–100 scale.

But substituting video-frame embeddings for text embeddings produced only 4.99 mtIoU on Charades-STA and 11.12 on QV-Highlights for the detailed LLM2CLIP configuration. The authors interpret this as evidence of a modality gap: image and text vectors can support cross-modal retrieval without being interchangeable inputs to a downstream model.

The text experiments also expose a temporal weakness. When frozen Qwen3-8B received descriptions directly, removing explicit frame indices reduced mtIoU from 45.47 to 13.82. Sequence order alone did not reliably tell it which positions an event occupied.

Replacing frames with nearby text concepts

The proposed remedy, Nearest Entity Replacement, or NER, uses a memory of 200,000 text embeddings representing nouns, verbs and action phrases extracted from training descriptions. For each frame, the system retrieves the nearest entities by cosine similarity, averages their embeddings and normalizes the result before the learned projection. No model parameters change during inference.

The authors argue that small semantic units reduce “concept bleed.” Retrieving a whole caption because one element matches a frame can also import unrelated objects or actions. Entity retrieval is intended to preserve relevant concepts without inheriting an entire scene description.

With detailed LLM2CLIP descriptions, NER raises mtIoU to 14.45 on Charades-STA and 20.72 on QV-Highlights. Caption-memory projection reaches 7.95 and 19.01 under the same configuration. Across the paper’s matched encoder and description comparisons, NER improves on both raw features and caption projection. The highest NER scores actually come from simplified LLM2CLIP descriptions: 15.57 and 22.29.

Partial transfer, with important qualifications

NER requires no video-based weight training, but its settings are not selected without video labels. The researchers tune the number of retrieved neighbors separately for each benchmark and configuration using 200 labelled training-split videos, then check the choice on 200 disjoint confirmation videos. That distinction matters when interpreting the paper’s zero-shot framing.

Evaluation protocols also constrain comparisons. QV-Highlights scores average over 2,701 query–reference-window rows from 1,486 queries, rather than choosing the best reference per query. Published video-trained results retain their original protocols and therefore provide context, not a controlled head-to-head comparison.

Finally, the authors report that NER locates broad event regions better than precise boundaries. Synthetic sequences may not reproduce real event timing, and evaluation on two video benchmarks leaves broader generalization unresolved. For AI development, the useful finding is narrower than replacing multimodal training: text can teach temporal behavior that partially transfers to video, while representation compatibility remains a major bottleneck.