MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence

Humans are born with vision-based 4D spatial-temporal intelligence, which enables us to perceive and reason about the evolution of 3D space over time from purely visual inputs. Despite its importance, this capability remains a significant bottleneck for current multimodal large language models (MLLMs). To tackle this challenge, we introduce MLLM-4D, a comprehensive framework designed to bridge the gaps in training data curation and model post-training for spatiotemporal understanding and reasoning. On the data front, we develop a cost-efficient data curation pipeline that repurposes existing stereo video datasets into high-quality 4D spatiotemporal instructional data. This results in the MLLM4D-2M and MLLM4D-R1-30k datasets for Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT), alongside MLLM4D-Bench for comprehensive evaluation. Regarding model training, our post-training strategy establishes a foundational 4D understanding via SFT and further catalyzes 4D reasoning capabilities by employing Group Relative Policy Optimization (GRPO) with specialized Spatiotemporal Chain of Thought (ST-CoT) prompting and Spatiotemporal reward functions (ST-reward) without involving the modification of architecture. Extensive experiments demonstrate that MLLM-4D achieves state-of-the-art spatial-temporal understanding and reasoning capabilities from purely 2D RGB inputs. Project page: https://github.com/GVCLab/MLLM-4D.

💡 Research Summary

MLLM‑4D tackles the long‑standing limitation of multimodal large language models (MLLMs) in perceiving and reasoning about the evolution of 3‑D space over time using only visual inputs. The authors first identify two bottlenecks: (1) the scarcity of large‑scale, high‑quality 4‑D instructional data, and (2) the lack of training recipes that endow existing MLLM architectures with 4‑D reasoning without architectural changes.

To address data scarcity, they devise an automated pipeline that repurposes existing stereo video datasets. For each frame they extract precise camera poses via Structure‑from‑Motion, metric depth from stereo matching, and per‑object 3‑D point clouds using GroundedSAM2 and PixelRefer. A video‑LLM (Gemini‑2.5‑flash) supplies fine‑grained semantic labels for each object. With this metadata they compute exact spatiotemporal relationships—absolute object distances, camera ego‑motion, and object‑camera dynamics—by applying physics‑based formulas (e.g., vector transformations, distance calculations). They then generate question‑answer pairs and a five‑step Spatiotemporal Chain‑of‑Thought (ST‑CoT) reasoning trace for each pair. The result is a 2‑million‑sample supervised fine‑tuning set (MLLM4D‑2M), a 30 k reinforcement‑learning set (MLLM4D‑R1‑30k), and a comprehensive benchmark (MLLM4D‑Bench) containing 6 k questions across six dynamic‑scene sub‑tasks.

Training proceeds in two stages. First, supervised fine‑tuning on MLLM4D‑2M teaches the model basic 4‑D concepts such as frame‑wise object and camera coordinates. Second, a Group Relative Policy Optimization (GRPO) reinforcement phase uses the ST‑CoT prompts to force the model to produce step‑by‑step reasoning. A novel Spatiotemporal Reward (ST‑reward) penalizes outputs that violate physical consistency, acting as a regularizer that discourages hallucinated motion. Importantly, the underlying MLLM architecture (e.g., Qwen‑VL, Gemini‑2.5) remains unchanged; only the post‑training regimen is altered.

Extensive experiments show that MLLM‑4D outperforms prior 3‑D‑focused MLLMs (e.g., VG‑LLM, Spatial‑MLLM) on all six benchmark sub‑tasks, achieving 8–12 percentage‑point gains in absolute and relative distance estimation, camera trajectory prediction, and object‑camera dynamics reasoning. It even reaches near‑human performance on the most complex “object‑camera dynamics” task (≈91 % accuracy). Moreover, the model retains competitive performance on static 3‑D QA benchmarks, indicating that 4‑D training improves general spatial reasoning.

The paper’s contributions are fourfold: (1) a cost‑effective, fully automated 4‑D data curation pipeline from stereo video, (2) large‑scale supervised and reinforcement datasets without manual annotation, (3) a GRPO‑based training framework that integrates ST‑CoT prompting and physics‑grounded ST‑reward to endow existing MLLMs with robust spatiotemporal reasoning, and (4) a comprehensive evaluation suite for dynamic scenes. Future work may extend the pipeline to monocular video, explore real‑time deployment in robotics or AR/VR, and integrate the 4‑D reasoning capability into interactive embodied agents.