Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Survey

In the last decade, Reinforcement Learning (RL) has achieved remarkable success in the control and decision-making of complex dynamical systems. However, most RL algorithms rely on the Markov Decision Process assumption, which is violated in practical cyber-physical systems affected by sensing delays, actuation latencies, and communication constraints. Such time delays introduce memory effects that can significantly degrade performance and compromise stability, particularly in networked and multi-agent environments. This paper presents a comprehensive survey of RL methods designed to address time delays in control systems. We first formalize the main classes of delays and analyze their impact on the Markov property. We then systematically categorize existing approaches into five major families: state augmentation and history-based representations, recurrent policies with learned memory, predictor-based and model-aware methods, robust and domain-randomized training strategies, and safe RL frameworks with explicit constraint handling. For each family, we discuss underlying principles, practical advantages, and inherent limitations. A comparative analysis highlights key trade-offs among these approaches and provides practical guidelines for selecting suitable methods under different delay characteristics and safety requirements. Finally, we identify open challenges and promising research directions, including stability certification, large-delay learning, multi-agent communication co-design, and standardized benchmarking. This survey aims to serve as a unified reference for researchers and practitioners developing reliable RL-based controllers in delay-affected cyber-physical systems.

💡 Research Summary

The paper provides a comprehensive survey of reinforcement learning (RL) methods that explicitly address time delays in control systems, a problem that undermines the Markov Decision Process (MDP) assumption underlying most RL algorithms. It begins by categorizing delays into four principal types: observation delays (τₒ), action/actuation delays (τₐ), state/communication delays in distributed or multi‑agent settings (τₛ), and stochastic delays including jitter and packet loss. For each type, the authors formalize how the delay introduces dependence on past states and actions, effectively turning the control problem into a non‑Markovian or partially observable process.

Building on this modeling foundation, the survey organizes existing literature into five methodological families:

1. State Augmentation / History‑Based Representations – By concatenating past observations and actions into an extended state vector (˜sₜ), the delayed system can be recast as a standard MDP (Delay‑Aware MDP). This approach restores the Markov property but suffers from dimensionality explosion and is limited to fixed‑length delays.

2. Recurrent Policies with Learned Memory – Recurrent neural networks (RNNs, LSTMs, GRUs) embed a learned memory that can capture variable‑length histories. They are flexible for stochastic or time‑varying delays, yet present challenges in training stability, interpretability, and computational overhead.

3. Predictor‑Based and Model‑Aware Strategies – These methods either learn a dynamics model or employ classical predictor‑based compensation (e.g., Smith predictors) to estimate the current state from delayed measurements. They enable theoretical stability analysis but are vulnerable to model mismatch and require additional system identification effort.

4. Robust and Domain‑Randomized Training Paradigms – By randomizing delay parameters during simulation, policies become robust to a range of latency conditions. This technique is practical for networked control where latency fluctuates, but may lead to overly conservative behavior if the randomization space is too broad.

5. Safe RL Frameworks with Explicit Constraint Handling – Safety is enforced through constrained optimization, shield functions, or Lyapunov‑based penalties, guaranteeing that policies respect safety limits even under delayed feedback. While essential for safety‑critical cyber‑physical systems, these methods increase algorithmic complexity and often require careful tuning of constraint parameters.

The authors present a detailed comparative matrix covering sample efficiency, scalability, implementation difficulty, and safety guarantees for each family. They derive practical design guidelines: for short, fixed delays, simple state augmentation is usually sufficient; for variable or stochastic delays, recurrent or robust approaches are preferable; in multi‑agent scenarios with communication latency, predictor‑based compensation combined with safe RL is recommended.

Finally, the survey identifies four major open challenges: (1) automated Lyapunov‑based stability certification for delay‑aware RL; (2) scalable learning algorithms capable of handling hundreds of delay steps and complex stochastic delay distributions; (3) joint communication‑control co‑design for multi‑agent systems, including event‑triggered messaging and topology optimization; and (4) the creation of standardized benchmarks and datasets that encompass a wide spectrum of delay phenomena. By consolidating theoretical insights, algorithmic categories, and practical recommendations, the paper serves as a unified reference for researchers and practitioners aiming to develop reliable RL‑based controllers in delay‑affected cyber‑physical environments.