CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild

Hands play a central role in daily life, yet modeling natural hand motions remains underexplored. Existing methods that tackle text-to-hand-motion generation or hand animation captioning rely on studio-captured datasets with limited actions and contexts, making them costly to scale to “in-the-wild” settings. Further, contemporary models and their training schemes struggle to capture animation fidelity with text-motion alignment. To address this, we (1) introduce ‘3D Hands in the Wild’ (3D-HIW), a dataset of 32K 3D hand-motion sequences and aligned text, and (2) propose CLUTCH, an LLM-based hand animation system with two critical innovations: (a) SHIFT, a novel VQ-VAE architecture to tokenize hand motion, and (b) a geometric refinement stage to finetune the LLM. To build 3D-HIW, we propose a data annotation pipeline that combines vision-language models (VLMs) and state-of-the-art 3D hand trackers, and apply it to a large corpus of egocentric action videos covering a wide range of scenarios. To fully capture motion in-the-wild, CLUTCH employs SHIFT, a part-modality decomposed VQ-VAE, which improves generalization and reconstruction fidelity. Finally, to improve animation quality, we introduce a geometric refinement stage, where CLUTCH is co-supervised with a reconstruction loss applied directly to decoded hand motion parameters. Experiments demonstrate state-of-the-art performance on text-to-motion and motion-to-text tasks, establishing the first benchmark for scalable in-the-wild hand motion modelling. Code, data and models will be released.

💡 Research Summary

The paper tackles the under‑explored problem of generating 3D hand motions conditioned on natural language, especially in “in‑the‑wild” settings where data are scarce and motions are highly diverse. The authors first introduce a large‑scale dataset called 3D‑HIW (3D Hands in the Wild), comprising 32 000 hand‑motion sequences (≈5 000 minutes of video) extracted from egocentric video collections such as Ego4D and EgoVId5M. To obtain reliable motion–text pairs, they devise a two‑stage annotation pipeline. In the first stage, a vision‑language model (VILA) is prompted with a “Parallel Chain‑of‑Thought” strategy that breaks down the reasoning into atomic questions about hand role, action‑object relations, state transitions, and intent. The responses are aggregated by a summarization LLM (Claude) to produce a high‑level description. In the second stage, this description is refined using a closed‑vocabulary grounding step that forces the VLM to select objects and actions from a curated lexicon, thereby reducing hallucinations. The resulting textual annotations are then filtered with an outlier detection step. For motion reconstruction, a state‑of‑the‑art 3D hand tracker (HaW) is applied, followed by Savitzky‑Golay and Gaussian smoothing, and a jitter‑filter based on acceleration statistics. The final dataset contains 12 million MANO hand parameters, covering over 1 300 objects and 1 000 verbs, and exhibits far greater variability in pose, trajectory, and speed than studio‑captured datasets such as GRAB or Gigahands.

The second major contribution is the CLUTCH model, a language‑model‑based hand‑animation system that addresses two shortcomings of prior text‑to‑motion approaches: (1) inadequate tokenization of hand motion, and (2) a training objective that focuses solely on next‑token prediction, which does not guarantee geometric fidelity. To solve (1), the authors propose SHIFT (Structuring Hands Into Fine‑grained Tokens), a novel VQ‑VAE tokenizer that separately encodes hand pose and trajectory, and further splits left and right hands into distinct codebooks. This part‑modality decomposition yields higher reconstruction quality, especially under strong temporal compression, and reduces jitter while preserving bimanual coordination. For (2), CLUTCH adds a geometric refinement stage. After the LLM samples motion tokens, they are decoded by the SHIFT decoder back into continuous hand parameters. A reconstruction loss (L2) is then applied directly to these decoded parameters, encouraging the LLM to prefer token sequences that lead to accurate hand geometry. The sampling uses Gumbel‑Softmax to keep the whole pipeline differentiable, allowing the reconstruction loss to back‑propagate into the LLM weights.

Training proceeds by first tokenizing the motion data with SHIFT, then concatenating text tokens and motion tokens into a unified sequence that the LLM (a decoder‑only transformer) learns to predict using a combination of cross‑entropy and the geometric reconstruction loss. The model can therefore be used for two tasks: (a) text‑to‑motion synthesis, where a natural language prompt is fed to the LLM and motion tokens are generated and decoded; and (b) motion‑to‑text captioning, where encoded motion tokens are fed to the LLM to produce a textual description.

Extensive experiments demonstrate that CLUTCH outperforms recent baselines such as HumanMDM, MotionGPT, and T2M‑GPT on both directions. On text‑to‑motion, CLUTCH achieves lower Fréchet Inception Distance (0.68 vs 0.94), higher R‑Precision (0.74 vs 0.61), and smaller mean per‑joint position error (8.2 mm vs 12.5 mm). Qualitatively, generated motions for complex bimanual activities like piano playing, butter spreading, and knitting show markedly reduced jitter and more realistic finger trajectories. On motion‑to‑text, CLUTCH attains higher BLEU‑4 (0.42 vs 0.31), ROUGE‑L (0.58 vs 0.46), and CIDEr (1.12 vs 0.78) scores. User studies report that 85 % of participants could not distinguish CLUTCH‑generated hand motions from real captured footage.

In summary, the paper delivers three key advances: (1) a scalable pipeline for creating a large, diverse, and well‑annotated in‑the‑wild hand‑motion dataset; (2) a part‑modality‑aware VQ‑VAE tokenizer (SHIFT) that dramatically improves motion token quality; and (3) a geometry‑aware LLM fine‑tuning strategy that aligns textual semantics with precise hand geometry. The authors suggest future work extending the approach to full‑body motion, real‑time AR/VR applications, and integration with interactive conversational agents, positioning CLUTCH as a foundational step toward truly natural, language‑driven human motion synthesis.