From Path Signatures to Sequential Modeling: Incremental Signature Contributions for Offline RL

Path signatures embed trajectories into tensor algebra and constitute a universal, non-parametric representation of paths; however, in the standard form, they collapse temporal structure into a single global object, which limits their suitability for decision-making problems that require step-wise reactivity. We propose the Incremental Signature Contribution (ISC) method, which decomposes truncated path signatures into a temporally ordered sequence of elements in the tensor-algebra space, corresponding to incremental contributions induced by last path increments. This reconstruction preserves the algebraic structure and expressivity of signatures, while making their internal temporal evolution explicit, enabling processing signature-based representations via sequential modeling approaches. In contrast to full signatures, ISC is inherently sensitive to instantaneous trajectory updates, which is critical for sensitive and stability-requiring control dynamics. Building on this representation, we introduce ISC-Transformer (ISCT), an offline reinforcement learning model that integrates ISC into a standard Transformer architecture without further architectural modification. We evaluate ISCT on HalfCheetah, Walker2d, Hopper, and Maze2d, including settings with delayed rewards and downgraded datasets. The results demonstrate that ISC method provides a theoretically grounded and practically effective alternative to path processing for temporally sensitive control tasks.

💡 Research Summary

The paper tackles a fundamental limitation of using path signatures in reinforcement learning: the standard truncated signature collapses an entire trajectory into a single global object, thereby losing sensitivity to recent observations and abrupt changes that are crucial for control tasks. To address this, the authors introduce Incremental Signature Contribution (ISC), a method that decomposes a truncated signature into a temporally ordered sequence of incremental components. Each component ΔS(k)ₙ at time step n captures the k‑th order contribution of the latest increment Δxₙ combined with the previously accumulated lower‑order signatures, following the recursive formula derived from Chen’s identity. This construction preserves the full algebraic information of the original signature while exposing its evolution over time, making it amenable to sequential models such as Transformers.

Building on ISC, the authors propose ISC‑Transformer (ISCT), an offline reinforcement‑learning architecture that integrates ISC tokens into a standard Decision‑Transformer pipeline with minimal changes. The input sequence consists of a Goal token (the sum of rewards within a fixed‑length window), the previous action, and for each timestep a triplet of State, INC (first‑order ISC) and CROSS (second‑order ISC) tokens. All tokens are linearly projected into a shared latent space, enriched with token‑type, channel, and positional embeddings. By feeding the incremental signature tokens alongside raw observations, the Transformer’s attention layers can directly attend to high‑order cross‑dimensional interactions and recent dynamics, improving both long‑range credit assignment and instantaneous reactivity.

Empirical evaluation spans four benchmark environments (HalfCheetah, Walker2d, Hopper, Maze2d) under standard offline RL settings as well as two challenging variations: delayed‑reward scenarios where rewards are shifted forward in time, and downgraded datasets with reduced coverage of the state‑action space. ISCT consistently outperforms strong baselines—including CQL, TD3+BC, IQL, and the original Decision‑Transformer—by 3–8% in average return on standard datasets and by 5–12% when rewards are delayed. Ablation studies reveal that models using only the global signature token suffer from poor sensitivity to local dynamics, confirming the advantage of the incremental formulation. The authors also discuss computational considerations: higher‑order ISC tensors grow exponentially, so they propose channel‑wise decomposition and limit practical usage to first and second order, which already yields substantial gains.

The paper’s contributions are threefold: (1) a theoretically grounded decomposition of path signatures that retains full information while exposing temporal granularity; (2) a straightforward integration of this representation into a Transformer‑based offline RL algorithm, demonstrating that existing architectures can be enhanced without redesign; and (3) extensive experiments showing robustness to delayed feedback and data scarcity. Limitations include the memory overhead of high‑order tensors and the focus on offline settings; future work may explore low‑rank approximations, online RL extensions, and multimodal signatures for richer observation spaces. Overall, the work bridges rough‑path theory and modern sequence modeling, offering a powerful new tool for temporally sensitive control problems.