Dynamic Reflections: Probing Video Representations with Text Alignment

The alignment of representations from different modalities has recently been shown to provide insights on the structural similarities and downstream capabilities of different encoders across diverse data types. While significant progress has been made in aligning images with text, the temporal nature of video data remains largely unexplored in this context. In this work, we conduct the first comprehensive study of video-text representation alignment, probing the capabilities of modern video and language encoders. Our findings reveal several key insights. First, we demonstrate that cross-modal alignment highly depends on the richness of both visual (static images vs. multi-frame videos) and text (single caption vs. a collection) data provided at test time, especially when using state-of-the-art video encoders. We propose parametric test-time scaling laws that capture this behavior and show remarkable predictive power against empirical observations. Secondly, we investigate the correlation between semantic alignment and performance on both semantic and non-semantic downstream tasks, providing initial evidence that strong alignment against text encoders may be linked to general-purpose video representation and understanding. Finally, we correlate temporal reasoning with cross-modal alignment providing a challenging test-bed for vision and language models. Overall, our work introduces video-text alignment as an informative zero-shot way to probe the representation power of different encoders for spatio-temporal data. Project page can be found at https://video-prh.github.io/

💡 Research Summary

Dynamic Reflections: Probing Video Representations with Text Alignment presents the first comprehensive study of cross‑modal alignment between modern video encoders and large language models (LLMs). Building on the Platonic Representation Hypothesis (PRH), which posits that sufficiently scaled neural networks converge toward a shared latent “Platonic” space, the authors ask whether this convergence extends from static image‑text pairs to the temporal domain of video‑text pairs.

Methodologically, the paper adapts the Mutual k‑Nearest Neighbor (MkNN) metric—originally used for image‑text alignment—to evaluate video‑text similarity. A test set of 1,024 video‑caption pairs is drawn from two public datasets (VATEX and the Perception Encoder Video Datasets, PVD). Each video is accompanied by up to ten human‑written English captions. Video embeddings are obtained from 121 video models, ranging from self‑supervised video MAE variants (VideoMAE‑base, ‑large, ‑huge) to vision‑transformer based models (ViViT, CLIP‑family, DINO‑family, MAE, etc.). Text embeddings are extracted from nine LLMs, including Gemma‑2‑9B‑IT, LLaMA‑7B, and Gemma‑7B.

The key experimental manipulation is test‑time data richness. For videos, the authors uniformly sample n_f frames (1, 16, 32, 64, 80) and average the resulting frame‑level representations, effectively turning a video into a multi‑frame “image” for the encoder. For text, they concatenate anywhere from a single caption to the full set of ten captions before feeding the string to the LLM, then average token‑level embeddings. This two‑dimensional scaling allows the authors to quantify how much additional visual and linguistic context improves alignment.

Results show a dramatic increase in MkNN alignment as both n_f and the number of captions grow. With a single frame and a single caption, alignment scores hover around 0.16–0.18, mirroring earlier image‑text findings. When 64–80 frames and ten captions are used, scores rise to 0.38–0.44 for many models, approaching the upper bound observed in static modalities. The authors fit a parametric scaling law A = A₀·n_f^α·c_i^β, where α≈0.42 and β≈0.18, achieving R² > 0.92 on held‑out data. This demonstrates that test‑time richness, not just model capacity, is a primary driver of cross‑modal similarity.

Beyond raw alignment, the paper investigates the relationship between alignment scores and downstream performance on three benchmark tasks: (1) Kinetics‑400 action classification, (2) Something‑Something‑V2 (SSv2) classification, and (3) video‑to‑text retrieval on the same datasets. Correlations are positive but clearly non‑linear. For instance, VideoMAE‑large, despite a moderate alignment of ~0.34, achieves top‑tier SSv2 accuracy (≈62 %). Conversely, DINOv2‑giant shows high alignment (~0.44) but lags in Kinetics‑400 retrieval (≈31 %). These findings suggest that while semantic alignment captures a useful aspect of representation quality, it does not fully predict performance on tasks that require fine‑grained temporal discrimination or motion‑specific cues.

To probe temporal reasoning, the authors construct a “temporal challenge” set where videos are either (a) temporally shuffled or (b) have causal action order reversed. They then measure alignment degradation relative to the original clips. Most models exhibit only modest drops, indicating that current MkNN‑based alignment is dominated by static visual semantics rather than true temporal understanding. This highlights a limitation: existing video‑text alignment metrics may overlook the causal structure that distinguishes video from a collection of frames.

The paper’s contributions can be summarized as follows:

1. Test‑time scaling laws – Demonstrates that increasing the number of sampled frames and captions yields predictable, quantifiable gains in video‑text alignment.
2. Zero‑shot probing metric – Positions MkNN alignment as a cheap, model‑agnostic proxy for assessing the semantic richness of video encoders without any fine‑tuning.
3. Non‑linear alignment‑performance relationship – Shows that strong alignment is neither necessary nor sufficient for downstream success, emphasizing the need for complementary evaluation.
4. Temporal reasoning benchmark – Provides a first‑order test of whether alignment captures motion and causality, revealing current gaps.

Limitations include the relatively small test set (1 024 pairs), reliance on English captions only, and the fact that MkNN alignment does not directly measure temporal dynamics. Future work could explore larger multilingual corpora, develop temporally aware alignment metrics (e.g., dynamic time warping of neighbor graphs), or integrate alignment objectives into pre‑training to encourage joint spatio‑temporal semantics.

Overall, Dynamic Reflections establishes video‑text alignment as a valuable, scalable probe for the latent structure of video models, extends the Platonic Representation Hypothesis into the temporal domain, and opens new avenues for evaluating and improving multimodal AI systems.