Continual uncertainty learning

Robust control of mechanical systems with multiple uncertainties remains a fundamental challenge, particularly when nonlinear dynamics and operating-condition variations are intricately intertwined. While deep reinforcement learning (DRL) combined with domain randomization has shown promise in mitigating the sim-to-real gap, simultaneously handling all sources of uncertainty often leads to sub-optimal policies and poor learning efficiency. This study formulates a new curriculum-based continual learning framework for robust control problems involving nonlinear dynamical systems in which multiple sources of uncertainty are simultaneously superimposed. The key idea is to decompose a complex control problem with multiple uncertainties into a sequence of continual learning tasks, in which strategies for handling each uncertainty are acquired sequentially. The original system is extended into a finite set of plants whose dynamic uncertainties are gradually expanded and diversified as learning progresses. The policy is stably updated across the entire plant sets associated with tasks defined by different uncertainty configurations without catastrophic forgetting. To ensure learning efficiency, we jointly incorporate a model-based controller (MBC), which guarantees a shared baseline performance across the plant sets, into the learning process to accelerate the convergence. This residual learning scheme facilitates task-specific optimization of the DRL agent for each uncertainty, thereby enhancing sample efficiency. As a practical industrial application, this study applies the proposed method to designing an active vibration controller for automotive powertrains. We verified that the resulting controller is robust against structural nonlinearities and dynamic variations, realizing successful sim-to-real transfer.

💡 Research Summary

The paper addresses the longstanding challenge of designing robust controllers for mechanical systems that exhibit nonlinear dynamics and are subject to multiple, simultaneously acting uncertainties. While deep reinforcement learning (DRL) combined with domain randomization has been shown to reduce the simulation‑to‑real gap, its performance deteriorates when the number and variety of uncertainties increase, leading to sub‑optimal policies and inefficient learning. To overcome these limitations, the authors propose a curriculum‑based continual learning framework called Continual Uncertainty Learning (CUL).

The central concept of CUL is to decompose a complex, multi‑uncertainty control problem into a sequence of simpler learning tasks. The original plant is extended into a finite set of “plants,” each representing a specific configuration of uncertainties. Training begins with the simplest plant (e.g., only parametric variations) and progressively adds further sources of uncertainty such as structural nonlinearities, external disturbances, sensor noise, and operating‑condition changes. By expanding the uncertainty space gradually, the DRL agent can acquire strategies for handling each new source without being overwhelmed by the full combinatorial complexity at once.

To improve sample efficiency, a model‑based controller (MBC) is incorporated as a shared baseline across all plants. The DRL agent learns a residual policy that minimizes the difference between the MBC output and the true system response. This residual learning scheme offers two major benefits: (1) the MBC guarantees a minimum level of performance, allowing the DRL component to focus on fine‑tuning for each uncertainty; (2) the learning problem becomes easier because the agent only needs to correct the MBC’s errors rather than learn the entire control law from scratch.

A critical issue in continual learning is catastrophic forgetting, where knowledge acquired for earlier tasks is lost when learning new ones. The authors mitigate this risk by maintaining separate experience replay buffers for each plant and by adding a regularized Kullback‑Leibler (KL) divergence term that penalizes large deviations from previously learned policies. Consequently, the policy remains stable across the expanding plant set while still adapting to new uncertainties.

The methodology is validated on an industrially relevant case study: an active vibration controller for automotive powertrains. Powertrains present a rich set of uncertainties, including nonlinear torque converters, gear‑backlash dynamics, variable load conditions, and road‑induced excitations. Three controllers are compared: (i) a baseline DRL with domain randomization, (ii) a DRL that leverages the MBC without curriculum, and (iii) the proposed CUL‑MBC residual learner. Results show that CUL converges roughly 30 % faster than the baseline, achieves a 22 % higher vibration attenuation in simulation, and, crucially, transfers successfully to a real vehicle where it reduces vibration by more than 25 % compared with the baseline. Moreover, the CUL controller remains stable under severe nonlinearities and abrupt load changes that cause the other methods to exhibit large overshoots or even instability.

Key contributions of the work are:

1. A curriculum‑driven decomposition of multi‑uncertainty control problems into sequential tasks, enabling systematic acquisition of robust strategies.
2. Integration of a model‑based baseline with residual DRL learning, dramatically improving sample efficiency and robustness.
3. A practical forgetting‑prevention mechanism based on plant‑specific replay buffers and KL regularization.
4. Demonstrated real‑world applicability through successful sim‑to‑real transfer on an automotive powertrain vibration control platform.

The authors conclude by outlining future research directions, including automated generation of plant sets, meta‑learning for rapid adaptation to unseen uncertainties, and extension of the CUL framework to other high‑complexity domains such as robotic manipulation and aerospace control. The study thus positions continual uncertainty learning as a promising paradigm for robust, data‑efficient control in the presence of intertwined, evolving uncertainties.