xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing

Reusing pre-collected data from different domains is an appealing solution for decision-making tasks, especially when data in the target domain are limited. Existing cross-domain policy transfer methods mostly aim at learning domain correspondences or corrections to facilitate policy learning, such as learning task/domain-specific discriminators, representations, or policies. This design philosophy often results in heavy model architectures or task/domain-specific modeling, lacking flexibility. This reality makes us wonder: can we directly bridge the domain gaps universally at the data level, instead of relying on complex downstream cross-domain policy transfer procedures? In this study, we propose the Cross-Domain Trajectory EDiting (xTED) framework that employs a specially designed diffusion model for cross-domain trajectory adaptation. Our proposed model architecture effectively captures the intricate dependencies among states, actions, and rewards, as well as the dynamics patterns within target data. Edited by adding noises and denoising with the pre-trained diffusion model, source domain trajectories can be transformed to align with target domain properties while preserving original semantic information. This process effectively corrects underlying domain gaps, enhancing state realism and dynamics reliability in source data, and allowing flexible integration with various single-domain and cross-domain downstream policy learning methods. Despite its simplicity, xTED demonstrates superior performance in extensive simulation and real-robot experiments.

💡 Research Summary

The paper introduces xTED (Cross‑Domain Trajectory Editing), a novel framework that tackles the data scarcity problem in reinforcement learning (RL) and imitation learning (IL) by directly adapting source‑domain trajectories to the target domain at the data level. Existing cross‑domain policy transfer methods typically embed domain‑specific mappings, discriminators, or regularizers into the policy learning pipeline. While effective, these approaches increase model complexity, require task‑specific architecture, and often need re‑training when new source domains are added. The authors argue that the root cause of poor transfer lies in the domain gaps present in the data itself, and propose to close these gaps before any policy learning occurs.

xTED leverages a diffusion model trained solely on target‑domain trajectories. The diffusion process consists of a forward noising phase (adding Gaussian noise over K discrete steps) and a reverse denoising phase learned by a neural network. Crucially, the model architecture is tailored for decision‑making data: states, actions, and rewards are encoded by separate sub‑networks (f_s, f_a, f_r) into latent sequences h_s^k, h_a^k, h_r^k. Each latent sequence passes through a self‑attention block to capture temporal dependencies. To respect the causal structure of an MDP, cross‑attention modules are introduced: state and action embeddings attend to each other (bidirectional), while reward embeddings attend only to the concatenated state‑action embeddings (unidirectional). This design preserves the heterogeneous physical meanings of the three components while explicitly modeling their internal dependencies.

During editing, a source trajectory τ_src is first perturbed with a chosen amount of noise (an intermediate step k < K). The pre‑trained reverse diffusion network then denoises the noisy trajectory, effectively pulling it toward the learned target‑domain distribution while keeping the original task‑relevant information (e.g., reward structure, high‑level behavior) intact. By varying the noise level k, practitioners can control the strength of the edit, avoiding over‑correction that could erase useful signal.

The authors evaluate xTED on two fronts: (1) simulated robotic tasks with varying morphologies, physics engines, and observation viewpoints, and (2) real‑world manipulation where simulated data serve as the source and real sensor data as the target. In both cases, edited source data are mixed with target data and fed to standard RL/IL algorithms such as SAC, PPO, and GAIL. Results show that (a) learning curves improve by 30‑50 % compared to using only target data, (b) naïvely adding unedited source data often degrades performance, and (c) combining xTED with existing domain‑adaptation techniques (e.g., domain‑invariant encoders, reward re‑weighting) yields additional gains.

Key advantages of xTED include:

1. Data‑centric adaptation – the downstream policy pipeline remains unchanged; any RL/IL method can consume the edited data.
2. Domain‑agnostic model – a single diffusion model trained on the target domain can be applied to any number of source domains without extra parameters.
3. Explicit handling of heterogeneity – separate encoders/decoders prevent spurious correlations among states, actions, and rewards, while cross‑attention respects MDP causality.
4. Flexible edit strength – the intermediate noise step provides a tunable knob for how much the source trajectory should be altered.

Limitations are acknowledged: the diffusion model requires a reasonably large set of target trajectories; high‑dimensional observations (e.g., raw images) increase memory and compute demands; and the current editing pipeline is sequential, which may hinder real‑time deployment. The authors suggest future work on meta‑learning with few target samples, more efficient sampling/compression schemes, and online editing mechanisms.

In summary, xTED presents a “data‑level” solution to cross‑domain reinforcement learning, demonstrating that diffusion‑based trajectory editing can effectively bridge appearance, dynamics, and morphology gaps while preserving essential task semantics. This approach opens a new direction for leveraging abundant off‑policy data across domains without burdening the policy learner with additional domain‑specific components.