Beyond the Patch: Exploring Vulnerabilities of Visuomotor Policies via Viewpoint-Consistent 3D Adversarial Object

Neural network-based visuomotor policies enable robots to perform manipulation tasks but remain susceptible to perceptual attacks. For example, conventional 2D adversarial patches are effective under fixed-camera setups, where appearance is relatively consistent; however, their efficacy often diminishes under dynamic viewpoints from moving cameras, such as wrist-mounted setups, due to perspective distortions. To proactively investigate potential vulnerabilities beyond 2D patches, this work proposes a viewpoint-consistent adversarial texture optimization method for 3D objects through differentiable rendering. As optimization strategies, we employ Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, exploiting distance-dependent frequency characteristics to induce textures effective across varying camera-object distances. We further integrate saliency-guided perturbations to redirect policy attention and design a targeted loss that persistently drives robots toward adversarial objects. Our comprehensive experiments show that the proposed method is effective under various environmental conditions, while confirming its black-box transferability and real-world applicability.

💡 Research Summary

The paper addresses a critical gap in the security of vision‑based robotic manipulation: existing 2‑D adversarial patches lose effectiveness when the camera viewpoint changes, which is common for wrist‑mounted (eye‑in‑hand) cameras. To overcome this limitation, the authors propose a viewpoint‑consistent adversarial attack that optimizes the texture of a 3‑D object rather than a flat patch. The attack is built on three technical pillars.

First, they employ Expectation over Transformation (EOT) to make the texture robust across a distribution of camera‑object poses. For each sampled transformation (distance r, azimuth θ, elevation ϕ), a short rollout of the target visuomotor policy is executed, the resulting action is evaluated with a custom loss, and the gradients are averaged over all samples. This ensures that the optimized texture works under the actual pose distribution encountered during execution.

Second, they introduce a Coarse‑to‑Fine (C2F) curriculum that respects the distance‑dependent frequency characteristics of visual perception. At large distances only low‑frequency (coarse) patterns are discernible, while at close range high‑frequency (fine) details become visible. The C2F schedule starts by optimizing coarse features using a beta‑distributed sampling that favors distant viewpoints, then gradually shifts the sampling toward nearer viewpoints to refine fine details. This staged optimization prevents conflicting objectives that would arise from trying to learn all frequencies simultaneously.

Third, they add a saliency‑guided component to redirect the policy’s visual attention. Using Grad‑CAM‑style gradients, a saliency map is computed for each policy forward pass. The loss maximizes average saliency over the adversarial object region while minimizing it over the true goal object region. The overall adversarial loss combines a pose‑driving term (orientation cosine similarity and Euclidean distance between the end‑effector’s intended next position and the adversarial object) with the saliency term, balanced by a weighting factor. To avoid gradient conflicts between the pose and saliency objectives, the Projecting Conflicting Gradients (PCGrad) algorithm is applied.

Because standard robot simulators use non‑differentiable rasterization, the authors devise a hybrid rendering pipeline. The whole scene is rendered with the simulator, but the adversarial object is rendered separately with a differentiable renderer (e.g., PyTorch3D). The two images are composited using a binary mask of the object, yielding a final image that can be back‑propagated through the policy network to obtain gradients with respect to the texture.

Experiments cover multiple manipulation tasks (reaching, grasping, obstacle avoidance) under varied lighting, background, and object placement conditions. Compared to 2‑D patches, the 3‑D adversarial object achieves 30‑50 % higher success rates when camera viewpoints change dramatically. The C2F curriculum contributes an additional 10‑15 % boost by ensuring robustness at both far and near distances. Transfer experiments show that textures optimized for one policy still degrade the performance of unseen policies (different network architectures or training data) by 20‑30 %, demonstrating black‑box applicability. Real‑world validation with a UR5 arm and a RealSense D435i camera confirms that 3‑D‑printed adversarial objects can reliably mislead the robot to reach the malicious object instead of the intended goal, even with variations in material and color.

The paper’s contributions are: (1) a systematic framework for viewpoint‑consistent 3‑D adversarial attacks on visuomotor policies, (2) a distance‑aware Coarse‑to‑Fine optimization strategy, (3) saliency‑guided loss to manipulate visual attention, (4) a hybrid differentiable rendering approach that integrates with existing simulators, and (5) extensive simulation and physical experiments demonstrating both white‑box and black‑box effectiveness. By exposing these vulnerabilities, the work paves the way for future defenses such as multi‑view training, texture randomization, or adversarially robust policy learning.