TwiFF (Think With Future Frames): A Large-Scale Dataset for Dynamic Visual Reasoning

Visual Chain-of-Thought (VCoT) has emerged as a promising paradigm for enhancing multimodal reasoning by integrating visual perception into intermediate reasoning steps. However, existing VCoT approaches are largely confined to static scenarios and struggle to capture the temporal dynamics essential for tasks such as instruction, prediction, and camera motion. To bridge this gap, we propose TwiFF-2.7M, the first large-scale, temporally grounded VCoT dataset derived from $2.7$ million video clips, explicitly designed for dynamic visual question and answer. Accompanying this, we introduce TwiFF-Bench, a high-quality evaluation benchmark of $1,078$ samples that assesses both the plausibility of reasoning trajectories and the correctness of final answers in open-ended dynamic settings. Building on these foundations, we propose the TwiFF model, a unified modal that synergistically leverages pre-trained video generation and image comprehension capabilities to produce temporally coherent visual reasoning cues-iteratively generating future action frames and textual reasoning. Extensive experiments demonstrate that TwiFF significantly outperforms existing VCoT methods and Textual Chain-of-Thought baselines on dynamic reasoning tasks, which fully validates the effectiveness for visual question answering in dynamic scenarios. Our code and data is available at https://github.com/LiuJunhua02/TwiFF.

💡 Research Summary

The paper addresses a critical gap in multimodal reasoning: existing Visual Chain‑of‑Thought (VCoT) methods are limited to static images and cannot handle temporal dynamics required for tasks such as instruction following, future prediction, or camera motion planning. To overcome this limitation, the authors introduce three major contributions.

First, they construct TwiFF‑2.7M, a large‑scale dataset of 2,708,318 dynamic VCoT instances derived from 2.7 million YouTube video clips (originally from the Panda‑70M collection). The data pipeline consists of three stages. In Stage 1, low‑quality or insufficiently dynamic clips are filtered out using four quantitative criteria: caption‑image semantic match (Unmasked Teacher score ≥ 0.43), visual desirability, minimum duration of 2 seconds, and a minimum optical‑flow magnitude (≥ 4). This reduces the raw pool to 10,596,462 high‑motion clips. In Stage 2, a multimodal large language model (InternVL‑3.5‑8B) classifies each clip into Instructional, Predictive, Camera, or Undesirable categories, discards the latter, and extracts at least two key frames per clip together with detailed textual narrations. After this step, 3,075,048 event instances remain. In Stage 3, the authors generate a VCoT chain for each event: the earliest key frame becomes the query image, a question is automatically generated conditioned on this frame, and a reasoning chain is built that alternates between textual reasoning about frame i‑1 and the generation of frame i, culminating in a final answer. This yields the TwiFF‑2.7M dataset, which is explicitly temporally grounded and covers three domains (instructional, predictive, camera).

Second, they release TwiFF‑Bench, a high‑quality evaluation benchmark consisting of 1,078 question‑answer pairs sampled from the test split of Panda‑70M, with zero overlap with training data. All samples are manually vetted to remove flawed reasoning or overly open‑ended responses. Evaluation is performed by GPT‑5.1 acting as a judge, which scores two dimensions separately: (1) CoT reasonableness (logical coherence, plausibility, and factual alignment with ground‑truth future frames) and (2) answer correctness. Scores range from 0 to 5, and the protocol explicitly avoids penalizing models that omit explicit image references, focusing instead on the quality of the reasoning trace.

Third, the authors propose the TwiFF model, a unified architecture that leverages pre‑trained video generation and image comprehension capabilities. Given the first key frame and a generated question, the model iteratively generates future frames (using a video diffusion or autoregressive generator) and interleaves them with textual reasoning steps. The visual cues are meant to ground the textual chain, while the textual context guides the generation of plausible future frames.

Extensive experiments demonstrate that TwiFF outperforms both static VCoT baselines (e.g., DeepEyes, SKETCHPAD) and textual CoT baselines (e.g., InternVL‑3.5) on TwiFF‑Bench, achieving gains of 1.8–2.3 points on the combined score. Ablation studies reveal that (a) the synergy between visual and textual modalities is essential—neither alone reaches the full performance; (b) physically plausible visual cues substantially boost answer accuracy (up to 12% improvement), whereas misleading cues degrade performance (≈ 8% drop); and (c) visual cues act as an information‑compression mechanism, filtering out irrelevant noise.

To assess out‑of‑distribution generalization, the model is evaluated on Seed‑Bench‑R1, a benchmark derived from EPIC‑Kitchens‑100 and Ego4D, containing 4,676 open‑ended predictive questions. Even without reference reasoning traces, TwiFF achieves higher answer accuracy than existing baselines, indicating robust generalization to unseen domains.

In summary, the paper delivers (1) the first large‑scale, temporally grounded VCoT dataset covering a broad spectrum of dynamic scenarios, (2) a rigorous benchmark that jointly evaluates reasoning plausibility and answer correctness, and (3) a unified model that demonstrates the effectiveness of interleaving future frame generation with textual chain‑of‑thought reasoning. The work sets a new standard for dynamic visual reasoning and opens avenues for future research on real‑time, embodied AI systems that must anticipate and plan over time.