VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?

The recent rapid advancement of Text-to-Video (T2V) generation technologies are engaging the trained models with more world model ability, making the existing benchmarks increasingly insufficient to evaluate state-of-the-art T2V models. First, current evaluation dimensions, such as per-frame aesthetic quality and temporal consistency, are no longer able to differentiate state-of-the-art T2V models. Second, event-level temporal causality-an essential property that differentiates videos from other modalities-remains largely unexplored. Third, existing benchmarks lack a systematic assessment of world knowledge, which are essential capabilities for building world models. To address these issues, we introduce VideoVerse, a comprehensive benchmark focusing on evaluating whether the current T2V model could understand complex temporal causality and world knowledge to synthesize videos. We collect representative videos across diverse domains and extract their event-level descriptions with inherent temporal causality, which are then rewritten into text-to-video prompts by independent annotators. For each prompt, we design ten evaluation dimensions covering dynamic and static properties, resulting in 300 prompts, 815 events, and 793 evaluation questions. Consequently, a human preference-aligned QA-based evaluation pipeline is developed by using modern vision-language models to systematically benchmark leading open- and closed-source T2V systems, revealing the current gap between T2V models and desired world modeling abilities.

💡 Research Summary

The paper introduces VideoVerse, a new benchmark designed to assess whether modern Text‑to‑Video (T2V) generators possess “world model” capabilities—that is, the ability to understand and apply physical laws, common sense, cultural knowledge, and temporal causality when synthesizing videos. The authors argue that existing benchmarks such as VBench, EvalCrafter, VBench2, and Video‑Bench focus primarily on frame‑level aesthetics, image fidelity, and explicit semantic alignment with fully specified prompts. As state‑of‑the‑art T2V models (e.g., CogVideoX, HunyuanVideo, StepVideo, and closed‑source systems like Veo3) have become increasingly powerful, these metrics are saturated and can no longer differentiate models that truly understand the underlying world.

VideoVerse addresses three key gaps: (1) it evaluates event‑level temporal causality, an essential property that distinguishes video from static media; (2) it systematically incorporates world knowledge, including natural constraints (physics, chemistry) and common sense (cultural facts); and (3) it adopts a “hidden semantics” prompt design, where only the initial condition is stated and the model must infer the unstated consequences (e.g., a rubber duck tossed onto a floor should bounce, reflecting elasticity and gravity).

The benchmark consists of 300 carefully curated prompts covering diverse domains, which together encode 815 events and 793 binary evaluation questions. For each prompt, ten evaluation dimensions are defined, split into five dynamic (Event Following, Mechanics, Interaction, Material Properties, Camera Control) and five static (Natural Constraints, Common Sense, Attribution Correctness, 2D Layout, 3D Depth). Dynamic dimensions require reasoning over temporal sequences, while static dimensions can be judged from a single frame. To obtain fine‑grained feedback, Mechanics, Interaction, and Material Properties are further broken down into sub‑questions (totaling 556 sub‑queries).

Evaluation is performed via a human‑preference‑aligned QA pipeline that leverages modern vision‑language models (VLMs). The VLM receives the generated video and the associated binary question, returning a yes/no answer that approximates a human evaluator. This approach minimizes human labor, ensures consistency, and enables large‑scale testing of both open‑source and closed‑source T2V systems.

Experimental results reveal a stark contrast between traditional quality metrics and world‑model performance. While models achieve near‑identical scores on FVD, IS, and FID, their VideoVerse scores diverge dramatically, especially on Event Following and Mechanics where the gap can exceed 40 %. Many models succeed at basic attribute correctness and camera control but fail to respect natural constraints (e.g., a lake at –20 °C remains liquid) or common sense (e.g., representing a “Japanese cultural tree” with a pine instead of a cherry blossom). The analysis shows that even the most advanced closed‑source systems still lack robust causal reasoning and physical plausibility.

The authors conclude that current T2V generators are far from true world models. They advocate for future research to focus on integrating structured physical and commonsense knowledge into diffusion‑based video generators, developing training objectives that reward causal consistency, and expanding evaluation frameworks like VideoVerse to cover richer multimodal interactions. By providing a comprehensive, discriminative benchmark, VideoVerse aims to guide the community toward T2V systems that not only look realistic but also reason about the world in a human‑like manner.