Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics

Reinforcement learning (RL) policies trained in simulation often suffer from severe performance degradation when deployed in real-world environments due to non-stationary dynamics. While Domain Randomization (DR) and meta-RL have been proposed to address this issue, they typically rely on extensive pretraining, privileged information, or high computational cost, limiting their applicability to real-time and edge systems. In this paper, we propose a lightweight online adaptation framework for RL based on Reservoir Computing. Specifically, we integrate an Echo State Networks (ESNs) as an adaptation module that encodes recent observation histories into a latent context representation, and update its readout weights online using Recursive Least Squares (RLS). This design enables rapid adaptation without backpropagation, pretraining, or access to privileged information. We evaluate the proposed method on CartPole and HalfCheetah tasks with severe and abrupt environment changes, including periodic external disturbances and extreme friction variations. Experimental results demonstrate that the proposed approach significantly outperforms DR and representative adaptive baselines under out-of-distribution dynamics, achieving stable adaptation within a few control steps. Notably, the method successfully handles intra-episode environment changes without resetting the policy. Due to its computational efficiency and stability, the proposed framework provides a practical solution for online adaptation in non-stationary environments and is well suited for real-world robotic control and edge deployment.

💡 Research Summary

This paper addresses the critical challenge of deploying reinforcement‑learning (RL) policies trained in simulation to real‑world systems where dynamics are non‑stationary. Existing remedies such as Domain Randomization (DR) and meta‑RL (e.g., Rapid Motor Adaptation) either require extensive pre‑training with privileged information (ground‑truth physics) or incur high computational cost during test‑time adaptation, making them unsuitable for resource‑constrained edge devices that need real‑time control.

The authors propose a lightweight online adaptation framework called ESN‑OA (Echo State Network‑based Online Adaptation). The core idea is to embed a fixed, randomly initialized recurrent reservoir (the ESN) into a Soft Actor‑Critic (SAC) agent. The reservoir processes the recent observation sequence and produces an internal state vector xₜ. Only the linear read‑out layer W_out, which maps xₜ to a prediction of the next state ŝₜ₊₁, is trained online using Recursive Least Squares (RLS). RLS minimizes a weighted sum of squared prediction errors with a forgetting factor λ, enabling rapid forgetting of outdated dynamics and fast convergence within a few time steps. The read‑out weights are initialized to zero and the inverse covariance matrix P₀ is set to a large diagonal value, ensuring high sensitivity at the start of deployment.

To make the predicted dynamics useful for control, the next‑state prediction ŝₜ₊₁ is concatenated with the current observation sₜ, forming an augmented state \tilde{s}_t =