Model-Based Data-Efficient and Robust Reinforcement Learning

A data-efficient learning-based control design method is proposed in this paper. It is based on learning a system dynamics model that is then leveraged in a two-level procedure. On the higher level, a simple but powerful optimization procedure is performed such that, for example, energy consumption in a vehicle can be reduced when hard state and action constraints are also introduced. Load disturbances and model errors are compensated for by a feedback controller on the lower level. In that regard, we briefly examine the robustness of both model-free and model-based learning approaches, and it is shown that the model-free approach greatly suffers from the inclusion of unmodeled dynamics. In evaluating the proposed method, it is assumed that a path is given, while the velocity and acceleration can be modified such that energy is saved, while still keeping speed limits and completion time. Compared with two well-known actor-critic reinforcement learning strategies, the suggested learning-based approach saves more energy and reduces the number of evaluated time steps by a factor of 100 or more.

💡 Research Summary

The paper proposes a data‑efficient, model‑based reinforcement learning (RL) framework for control design, targeting applications such as eco‑driving and robotic logistics where a path is predetermined but the velocity and acceleration profiles can be tuned to reduce energy consumption while respecting hard state and input constraints (speed limits, acceleration bounds, and a prescribed completion time). The approach is organized into two distinct layers.

In the upper layer, a simple physics‑informed system model is first identified from a limited amount of experimental data. The model may be linear (ARX, state‑space) or nonlinear (gray‑box regression), and the identification uses classic least‑squares or subspace methods. Once the model is available, a “temporal optimization” problem is formulated: the reference velocity trajectory is optimized to minimize an energy‑related cost function subject to the hard constraints. Because the path is fixed, the optimization reduces to a constrained trajectory‑planning problem that can be solved efficiently with standard convex or nonlinear programming tools.

The lower layer consists of a feedback controller that compensates for model errors, external disturbances, and unmodeled high‑frequency dynamics. The feedback can be a simple PI/PID, an LQR based on the identified linear model, or a more sophisticated nonlinear controller. By separating feed‑forward (optimal reference generation) from feedback (robustness), the method becomes modular: the model can be inspected and refined independently, the optimal trajectory can be recomputed without redesigning the feedback law, and the feedback can be retuned without re‑identifying the model.

To assess the benefits, the authors compare their modular model‑based RL against two well‑known model‑free actor‑critic algorithms: Twin‑Delayed DDPG (TD3) and Soft Actor‑Critic (SAC). Both model‑free methods require a carefully crafted reward that blends energy, time, and safety terms, and they must learn the Q‑function and policy simultaneously, which leads to high sample complexity. In the experiments (electric truck and warehouse robot), the model‑free approaches needed on the order of 10⁶ simulation steps to converge, whereas the proposed method required roughly 10⁴ steps—a reduction by a factor of 100 or more. Moreover, the model‑based method achieved 15–20 % additional energy savings compared with the best model‑free policy, while always respecting the imposed constraints.

A robustness analysis is also presented. By deliberately adding high‑frequency resonances (short time constants) that are omitted from the identified model, the authors show that model‑free policies become unstable or severely detuned, confirming earlier theoretical findings that model‑free RL is highly sensitive to unmodeled dynamics. In contrast, the feedback controller in the model‑based scheme can absorb these dynamics, preserving stability and performance.

Overall, the paper demonstrates that a two‑level, modular model‑based RL architecture provides superior data efficiency, easier constraint handling, and greater robustness to model mismatch than conventional model‑free deep RL. Future work is outlined to extend the approach to richer nonlinear model classes, multi‑vehicle coordination, and online adaptive model updating.