RISE-Video: Can Video Generators Decode Implicit World Rules?

While generative video models have achieved remarkable visual fidelity, their capacity to internalize and reason over implicit world rules remains a critical yet under-explored frontier. To bridge this gap, we present RISE-Video, a pioneering reasoning-oriented benchmark for Text-Image-to-Video (TI2V) synthesis that shifts the evaluative focus from surface-level aesthetics to deep cognitive reasoning. RISE-Video comprises 467 meticulously human-annotated samples spanning eight rigorous categories, providing a structured testbed for probing model intelligence across diverse dimensions, ranging from commonsense and spatial dynamics to specialized subject domains. Our framework introduces a multi-dimensional evaluation protocol consisting of four metrics: \textit{Reasoning Alignment}, \textit{Temporal Consistency}, \textit{Physical Rationality}, and \textit{Visual Quality}. To further support scalable evaluation, we propose an automated pipeline leveraging Large Multimodal Models (LMMs) to emulate human-centric assessment. Extensive experiments on 11 state-of-the-art TI2V models reveal pervasive deficiencies in simulating complex scenarios under implicit constraints, offering critical insights for the advancement of future world-simulating generative models.

💡 Research Summary

The paper introduces RISE‑Video, a novel benchmark designed to assess the ability of Text‑Image‑to‑Video (TI2V) generative models to internalize and reason over implicit world rules, moving beyond the conventional focus on visual fidelity and temporal coherence. RISE‑Video consists of 467 carefully curated samples, each annotated by human experts, and is organized into eight reasoning dimensions: Experiential, Perceptual, Temporal, Spatial, Commonsense, Societal, Subject‑specific, and Logical. Each dimension is further broken down into sub‑categories (e.g., physical commonsense, life commonsense, health‑care within Commonsense; physics, chemistry, geography, sports within Subject‑specific; view‑point, object arrangement, structural inference within Spatial, etc.), providing a comprehensive coverage of the knowledge required to generate videos that obey hidden constraints.

To evaluate models, the authors propose four complementary metrics:

1. Reasoning Alignment – measures whether the generated video correctly reflects the intended knowledge. For each sample, a set of manually crafted, knowledge‑aware binary questions is posed to a Large Multimodal Model (LMM) judge, which answers Yes/No. Scores are aggregated into a 0‑1 alignment value.
2. Temporal Consistency – assesses the coherence of motion over time. Different frame‑sampling rates (e.g., 2 fps for full‑progression events, lower rates for terminal‑state tasks) are used to balance evaluation cost and information sufficiency. Scores range from 1 to 5.
3. Physical Rationality – evaluates adherence to real‑world physics, biology, chemistry, etc., also on a 1‑5 scale.
4. Visual Quality – captures traditional perceptual quality (sharpness, realism) on a 1‑3 scale.

A key contribution is an automated evaluation pipeline that leverages LMMs as judges, dramatically reducing the need for large‑scale human labeling while maintaining high reliability. For tasks where linguistic description is insufficient (e.g., maze navigation, symmetry generation), the pipeline incorporates non‑linguistic verification methods such as color‑matching trajectory tracking or grid‑level positional alignment, ensuring that the LMM’s limitations do not compromise the assessment.

The authors benchmark 11 state‑of‑the‑art TI2V models—including CogVideoX, Sora, HunyuanVideo, and others—using the proposed pipeline. Results reveal a consistent pattern: while visual quality scores are relatively high (most models achieve 3–4 out of 3), Reasoning Alignment scores are low (often below 0.3), especially in Experiential, Societal, and Logical categories. Physical Rationality is also modest, indicating frequent violations of basic physical laws. Temporal Consistency varies, with short‑term tasks performing better than medium‑ or long‑term reasoning.

To validate the automated scores, the authors compare LMM judgments with human annotations on the same 467 samples, obtaining a Cohen’s κ of approximately 0.78, demonstrating strong agreement and confirming that the LMM‑as‑judge approach is a viable, cost‑effective alternative for large‑scale evaluation.

The paper’s significance lies in shifting the evaluation paradigm for video generation from pure aesthetics toward cognitive reasoning. By exposing the current models’ deficiencies in handling implicit constraints, RISE‑Video highlights several avenues for future research: integrating large‑scale multimodal commonsense knowledge during pre‑training, designing loss functions that penalize rule violations, developing dedicated reasoning modules that feed into video decoders, and expanding automated evaluation to multilingual and culturally diverse contexts.

In summary, RISE‑Video provides the first comprehensive, multi‑dimensional benchmark for probing the hidden reasoning capabilities of TI2V systems, offers a scalable LMM‑based evaluation pipeline with proven human alignment, and delivers actionable insights that can guide the next generation of world‑simulating generative video models.