A Practical Guide for Incorporating Symmetry in Diffusion Policy

Recently, equivariant neural networks for policy learning have shown promising improvements in sample efficiency and generalization, however, their wide adoption faces substantial barriers due to implementation complexity. Equivariant architectures typically require specialized mathematical formulations and custom network design, posing significant challenges when integrating with modern policy frameworks like diffusion-based models. In this paper, we explore a number of straightforward and practical approaches to incorporate symmetry benefits into diffusion policies without the overhead of full equivariant designs. Specifically, we investigate (i) invariant representations via relative trajectory actions and eye-in-hand perception, (ii) integrating equivariant vision encoders, and (iii) symmetric feature extraction with pretrained encoders using Frame Averaging. We first prove that combining eye-in-hand perception with relative or delta action parameterization yields inherent SE(3)-invariance, thus improving policy generalization. We then perform a systematic experimental study on those design choices for integrating symmetry in diffusion policies, and conclude that an invariant representation with equivariant feature extraction significantly improves the policy performance. Our method achieves performance on par with or exceeding fully equivariant architectures while greatly simplifying implementation.

💡 Research Summary

This paper addresses the practical challenge of incorporating symmetry into diffusion‑based robotic manipulation policies without resorting to heavyweight equivariant network designs. While fully equivariant architectures have demonstrated impressive gains in sample efficiency and generalization, their mathematical complexity and specialized layers make them difficult to adopt in modern policy pipelines such as Diffusion Policy. The authors propose three complementary, low‑overhead strategies that together achieve performance on par with—or even exceeding—state‑of‑the‑art equivariant models.

First, they analyze how an eye‑in‑hand camera combined with relative or delta trajectory action representations yields inherent SE(3) invariance. By expressing future gripper poses relative to the current gripper frame (relative trajectory) or as incremental motions in the moving local frame (delta trajectory), the action sequence does not change under any global SE(3) transformation of the world. Because the eye‑in‑hand image is also unchanged under such transformations, the overall observation–action mapping becomes SE(3)‑equivariant by construction. The paper formalizes absolute, relative, and delta trajectories, proves that relative and delta actions are invariant, and shows that a policy built on these representations automatically generalizes across unseen object poses without additional data augmentation.

Second, the authors recognize that pure invariant representations, while theoretically sound, fall short of the empirical performance of fully equivariant policies. To bridge this gap they introduce an equivariant vision encoder that sits in front of the standard diffusion denoising network. The encoder is designed to be equivariant to a discrete rotation group C_u (e.g., cyclic permutations of feature channels), thereby extracting symmetry‑aware visual features while leaving the diffusion backbone unchanged. This modular insertion preserves the simplicity of existing diffusion implementations yet supplies the richer local geometric cues that equivariant networks provide.

Third, they propose a “Frame Averaging” technique to retrofit pretrained, non‑equivariant encoders (such as ResNet or ViT) into a symmetry‑aware pipeline. Multiple transformed copies of the eye‑in‑hand image are passed through the pretrained encoder; their outputs are then averaged (or voted) to produce a single feature vector that is invariant to the applied group transformations. This approach leverages the expressive power of large‑scale vision models without requiring any architectural changes, and it can be combined with the invariant action representation for a fully symmetry‑consistent policy.

Extensive experiments on the MimicGen benchmark and other manipulation tasks compare (i) the baseline Diffusion Policy, (ii) a fully equivariant diffusion model, (iii) the invariant representation with an equivariant encoder, and (iv) the invariant representation with frame‑averaged pretrained encoders. Results show that the invariant + equivariant‑encoder configuration consistently outperforms the baseline and matches the fully equivariant model, while the frame‑averaging variant achieves comparable performance with far less implementation effort. Notably, using only a single eye‑in‑hand camera, the proposed method reaches or surpasses prior work that relied on multiple external cameras and 3‑D voxel inputs.

In summary, the paper demonstrates that symmetry can be effectively injected into diffusion policies through three practical avenues: (1) adopting SE(3)‑invariant observation and action formats, (2) augmenting the visual front‑end with an equivariant encoder, and (3) applying frame averaging to existing pretrained encoders. This modular recipe lowers the barrier to entry for researchers and practitioners who wish to exploit symmetry, offering a clear path to high‑performing, sample‑efficient policies without the overhead of designing end‑to‑end equivariant networks.