AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Models

Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic perception and control, yet most existing approaches primarily rely on VLM trained using 2D images, which limits their spatial understanding and action grounding in complex 3D environments. To address this limitation, we propose a novel framework that integrates depth estimation into VLA models to enrich 3D feature representations. Specifically, we employ a depth estimation baseline called VGGT to extract geometry-aware 3D cues from standard RGB inputs, enabling efficient utilization of existing large-scale 2D datasets while implicitly recovering 3D structural information. To further enhance the reliability of these depth-derived features, we introduce a new module called action assistant, which constrains the learned 3D representations with action priors and ensures their consistency with downstream control tasks. By fusing the enhanced 3D features with conventional 2D visual tokens, our approach significantly improves the generalization ability and robustness of VLA models. Experimental results demonstrate that the proposed method not only strengthens perception in geometrically ambiguous scenarios but also leads to superior action prediction accuracy. This work highlights the potential of depth-driven data augmentation and auxiliary expert supervision for bridging the gap between 2D observations and 3D-aware decision-making in robotic systems.

💡 Research Summary

**
The paper addresses a fundamental limitation of current Vision‑Language‑Action (VLA) models: their reliance on 2‑D visual features inherited from large‑scale vision‑language models (VLMs) hampers spatial reasoning and action grounding in complex three‑dimensional environments. To overcome this, the authors propose AugVLA‑3D, a framework that injects depth‑derived geometric cues into a VLA backbone while preserving compatibility with existing 2‑D pre‑training data.

The pipeline begins with a state‑of‑the‑art monocular depth estimator called VGGT. Given a standard RGB observation (or a short sequence of viewpoints), VGGT predicts a dense depth map. Using known camera intrinsics, the depth map is back‑projected into a point cloud. Because raw point clouds can be very large, a sampling operator reduces the number of points to a manageable size (e.g., 2 K points). These sampled points are then processed by a lightweight PointNet encoder, which outputs compact 3‑D feature vectors (f₃ᴰ) that capture local geometry and global spatial layout.

Simply concatenating f₃ᴰ with the 2‑D visual tokens, however, can destabilize the pretrained VLA representation. To align the new geometric information with the task‑specific action space, the authors introduce an “Action Assistant” module. This auxiliary head mirrors the primary Action Expert but contains far fewer parameters. It receives the PointNet features, generates intermediate action embeddings at each transformer layer, and injects them back into the corresponding layers of the main action head via a learnable scalar gate α(l) and a lightweight projection or cross‑attention transform T(·). The update rule ˜h(l)=hᵒʳⁱᵍ(l)+α(l)·T(hᵃˣ(l),f₃ᴰ) ensures that the 3‑D cues act as a regularizer, guiding the network toward geometrically consistent actions without overwhelming the pretrained semantic knowledge.

Experiments compare AugVLA‑3D against three baselines: (1) Gr00t, a pure 2‑D VLA model; (2) PointVLA, which injects LiDAR‑derived point clouds; and (3) a variant of the authors’ architecture without the Action Assistant. Benchmarks span manipulation tasks that require fine‑grained spatial reasoning, such as object stacking, collision avoidance, and reachability under occlusion. Results show that AugVLA‑3D improves success rates by 10–15 % over Gr00t in depth‑ambiguous scenes and outperforms PointVLA by 5–9 % despite not using any dedicated 3‑D sensor. Importantly, the additional computational overhead is modest (≈10 % increase in FLOPs), preserving real‑time feasibility for robot control.

The paper’s contributions are threefold: (1) a sensor‑free depth‑driven 3‑D feature extraction pipeline that leverages abundant 2‑D datasets; (2) a compact PointNet encoder that transforms back‑projected point clouds into task‑relevant embeddings; (3) the Action Assistant regularizer that aligns geometric features with action objectives while keeping the parameter budget low.

Limitations are acknowledged. Monocular depth estimation can suffer from lighting variations, reflective surfaces, or transparent objects, which may degrade point‑cloud quality. PointNet, while efficient, may struggle with highly complex geometries that require richer 3‑D encoders. The gating mechanism α(l) needs careful tuning; if mis‑learned, it could either suppress useful geometric cues or inject noisy signals. Future work is suggested to explore multi‑view depth fusion, dynamic point‑cloud processing, attention‑based 3‑D/2‑D fusion layers, and more expressive 3‑D backbones such as sparse convolutions.

In summary, AugVLA‑3D demonstrates that depth‑driven data augmentation combined with an auxiliary expert can effectively bridge the gap between 2‑D visual language models and the 3‑D spatial reasoning required for robust robot manipulation, offering a scalable and sensor‑agnostic path toward more capable embodied AI systems.