MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies

Rigid bodies constitute the smallest manipulable elements in the real world, and understanding how they physically interact is fundamental to embodied reasoning and robotic manipulation. Thus, accurate detection, segmentation, and tracking of moving rigid bodies is essential for enabling reasoning modules to interpret and act in diverse environments. However, current segmentation models trained on semantic grouping are limited in their ability to provide meaningful interaction-level cues for completing embodied tasks. To address this gap, we introduce MotionBit, a novel concept that, unlike prior formulations, defines the smallest unit in motion-based segmentation through kinematic spatial twist equivalence, independent of semantics. In this paper, we contribute (1) the MotionBit concept and definition, (2) a hand-labeled benchmark, called MoRiBo, for evaluating moving rigid-body segmentation across robotic manipulation and human-in-the-wild videos, and (3) a learning-free graph-based MotionBits segmentation method that outperforms state-of-the-art embodied perception methods by 37.3% in macro-averaged mIoU on the MoRiBo benchmark. Finally, we demonstrate the effectiveness of MotionBits segmentation for downstream embodied reasoning and manipulation tasks, highlighting its importance as a fundamental primitive for understanding physical interactions.

💡 Research Summary

The paper tackles a fundamental gap in visual perception for embodied agents: the ability to segment moving rigid bodies—the smallest manipulable physical units—independently of semantic class labels. Existing segmentation models excel at grouping pixels by human‑defined categories (e.g., “desk”, “keyboard”), but they provide little insight into how objects physically interact, which is crucial for tasks such as dexterous manipulation, robot planning, and embodied reasoning. To fill this void, the authors introduce the concept of a “MotionBit”, defined mathematically through kinematic spatial twist equivalence.

A MotionBit is the set of all image points that share an identical, non‑zero spatial twist over a chosen observation window. The spatial twist V_s =