ESPADA: Execution Speedup via Semantics Aware Demonstration Data Downsampling for Imitation Learning

Behavior-cloning based visuomotor policies enable precise manipulation but often inherit the slow, cautious tempo of human demonstrations, limiting practical deployment. However, prior studies on acceleration methods mainly rely on statistical or heuristic cues that ignore task semantics and can fail across diverse manipulation settings. We present ESPADA, a semantic and spatially aware framework that segments demonstrations using a VLM-LLM pipeline with 3D gripper-object relations, enabling aggressive downsampling only in non-critical segments while preserving precision-critical phases, without requiring extra data or architectural modifications, or any form of retraining. To scale from a single annotated episode to the full dataset, ESPADA propagates segment labels via Dynamic Time Warping (DTW) on dynamics-only features. Across both simulation and real-world experiments with ACT and DP baselines, ESPADA achieves approximately a 2x speed-up while maintaining success rates, narrowing the gap between human demonstrations and efficient robot control.

💡 Research Summary

The paper tackles a fundamental bottleneck in vision‑based behavior‑cloning (BC) policies for robot manipulation: the policies inherit the slow, cautious tempo of human demonstrations, which limits their practical deployment in time‑critical settings. Existing acceleration techniques rely on statistical heuristics (e.g., uniform downsampling, velocity scaling) that ignore the semantic structure of a task and therefore often degrade performance when applied across diverse manipulation scenarios.

ESPADA (Execution Speedup via Semantics‑Aware Demonstration Data Downsampling) introduces a semantic‑ and spatially‑aware pipeline that automatically identifies which portions of a demonstration are “precision‑critical” and which are “non‑critical.” The system consists of three main components.

1. Semantic Segmentation via VLM‑LLM – A visual‑language model (VLM) processes RGB‑D streams to extract 3‑D gripper‑object relations such as contact, distance, and relative orientation. These relational descriptors are fed to a large language model (LLM) that translates them into natural‑language segment labels (e.g., “approach,” “grasp,” “move,” “insert”). The resulting labels partition a demonstration into a sequence of semantically meaningful phases.

2. Aggressive, Phase‑Specific Downsampling – For phases identified as non‑critical (e.g., free‑space motion, coarse transport), ESPADA applies aggressive temporal compression, dropping frames at a factor of 2–4 while preserving the underlying kinematic trajectory. For precision‑critical phases (e.g., fine alignment, insertion), the original frame rate is retained, ensuring that the BC policy still sees the high‑resolution data it needs to maintain accuracy. Importantly, this downsampling is performed as a preprocessing step; the BC network architecture, weights, and loss functions remain untouched, and no additional training is required.

3. Dataset‑Wide Label Propagation via DTW – Manually labeling every episode is infeasible. ESPADA therefore annotates a single reference episode per task and propagates those labels to the rest of the dataset using Dynamic Time Warping (DTW) on dynamics‑only features (joint angles, velocities, torques). DTW aligns sequences of differing lengths, allowing the system to map the semantic phases from the reference to any new demonstration based solely on motion similarity. This yields consistent segment labels across the entire dataset with minimal computational overhead.

The authors evaluate ESPADA on both simulation (Isaac Gym) and real‑world hardware (UR5e robot with RG2 gripper) across six manipulation tasks: pick‑and‑place, insertion, rotation, stacking, tool exchange, and fine adjustment. Two state‑of‑the‑art BC baselines are used: ACT (Attention‑based Cloning Transformer) and DP (Diffusion Policy). Results show that ESPADA consistently halves the execution time (≈48 % reduction) while preserving success rates within 1–2 % of the original policies. In some precision‑heavy tasks, the reduced jitter from faster motion even yields a modest increase in success.

Key contributions are: (i) a VLM‑LLM pipeline that automatically extracts semantically meaningful phases from raw visual demonstrations; (ii) a phase‑aware downsampling strategy that accelerates execution without sacrificing task‑critical accuracy; (iii) a lightweight DTW‑based label propagation method that scales the semantic segmentation from a single annotated episode to an entire dataset; and (iv) a demonstration that these techniques can be applied to existing BC pipelines without any architectural changes or extra training data.

Limitations are acknowledged. The quality of the semantic segmentation depends on the VLM‑LLM’s ability to correctly infer 3‑D relations, which can be challenged by occlusions, reflective or transparent objects. DTW‑based propagation assumes that dynamics‑only features are sufficiently discriminative; in highly redundant or noisy motion data the alignment may become costly or ambiguous. Future work is suggested to improve multimodal attention mechanisms for more robust relation extraction and to explore more efficient time‑series alignment algorithms (e.g., soft‑DTW, learned warping functions) to reduce computational load.

In summary, ESPADA offers a practical, plug‑and‑play solution that bridges the gap between human‑like demonstration quality and robot‑level execution efficiency. By leveraging task semantics and spatial context, it enables robots to operate at speeds comparable to engineered controllers while retaining the adaptability and precision of imitation‑learning policies, thereby advancing the feasibility of real‑time, high‑throughput robot manipulation in industrial and collaborative settings.