Reinforcement learning for port-Hamiltonian systems

Passivity-based control (PBC) for port-Hamiltonian systems provides an intuitive way of achieving stabilization by rendering a system passive with respect to a desired storage function. However, in most instances the control law is obtained without any performance considerations and it has to be calculated by solving a complex partial differential equation (PDE). In order to address these issues we introduce a reinforcement learning approach into the energy-balancing passivity-based control (EB-PBC) method, which is a form of PBC in which the closed-loop energy is equal to the difference between the stored and supplied energies. We propose a technique to parameterize EB-PBC that preserves the systems’s PDE matching conditions, does not require the specification of a global desired Hamiltonian, includes performance criteria, and is robust to extra non-linearities such as control input saturation. The parameters of the control law are found using actor-critic reinforcement learning, enabling learning near-optimal control policies satisfying a desired closed-loop energy landscape. The advantages are that near-optimal controllers can be generated using standard energy shaping techniques and that the solutions learned can be interpreted in terms of energy shaping and damping injection, which makes it possible to numerically assess stability using passivity theory. From the reinforcement learning perspective, our proposal allows for the class of port-Hamiltonian systems to be incorporated in the actor-critic framework, speeding up the learning thanks to the resulting parameterization of the policy. The method has been successfully applied to the pendulum swing-up problem in simulations and real-life experiments.

💡 Research Summary

The paper introduces a novel framework that merges energy‑balancing passivity‑based control (EB‑PBC) for port‑Hamiltonian (PH) systems with model‑free reinforcement learning, specifically an actor‑critic algorithm. Traditional passivity‑based control designs a desired Hamiltonian and solves a set of partial differential equations (PDEs) to shape the system’s energy and inject damping. While this approach guarantees stability through passivity, it suffers from three major drawbacks: (1) the need to pre‑specify a global desired Hamiltonian, (2) the analytical or numerical difficulty of solving the matching PDEs, and (3) the lack of explicit performance criteria such as fast convergence or low control effort.

To overcome these limitations, the authors propose to parameterize the EB‑PBC law in a way that inherently satisfies the PDE matching conditions. The desired storage function (H_d(x;\theta)) and the damping matrix (R_d(x;\theta)) are expressed as functions of a parameter vector (\theta) (e.g., linear combinations of basis functions or neural networks). The control input then takes the classic EB‑PBC form (u = \beta(x;\theta) + \alpha(x;\theta)), but the parameters (\theta) are not fixed a priori; they are learned online.

A reinforcement‑learning objective is built on a reward that combines (i) an energy‑based term that penalizes the deviation between the current Hamiltonian and a desired energy landscape, and (ii) conventional performance terms such as state‑tracking error, control‑magnitude penalty, and a penalty for violating input saturation limits. By embedding the input‑saturation penalty directly into the reward, the learned policy automatically respects actuator constraints.

The learning algorithm follows the actor‑critic paradigm. The critic approximates the value function (V^\pi(x)) using temporal‑difference (TD) errors that are computed with the exact PH dynamics, exploiting the known structure of the Hamiltonian and interconnection matrix. This structure‑aware TD error yields more accurate gradient estimates than generic model‑free methods. The actor updates the policy parameters (\theta) by ascending the policy gradient supplied by the critic, while remaining inside the subspace defined by the PDE‑matching constraints. Consequently, every policy generated during learning is guaranteed to be a valid EB‑PBC law, preserving passivity and the associated Lyapunov‑based stability guarantees.

The methodology is validated on the classic pendulum swing‑up problem. In simulation, the learned controller achieves a significantly shorter swing‑up time and reduces the integrated control effort compared with a hand‑designed EB‑PBC controller that uses a fixed desired Hamiltonian. Real‑world experiments on an actual pendulum hardware confirm that the same learned parameters can be transferred without retuning, and the controller successfully handles actuator saturation while still achieving reliable swing‑up.

Key contributions of the work are:

1. Structure‑preserving parameterization – the policy space is constrained to satisfy the PH PDEs, guaranteeing passivity for any learned policy.
2. Performance‑oriented reward design – desired closed‑loop energy shapes and practical performance metrics are incorporated directly into the reinforcement‑learning objective, eliminating the need for an explicit global Hamiltonian.
3. Efficient learning – by leveraging the known PH dynamics in the critic’s TD error, the actor‑critic algorithm converges faster and more reliably than generic model‑free RL approaches.
4. Experimental verification – both simulated and physical pendulum experiments demonstrate that the approach yields near‑optimal controllers that are robust to nonlinearities such as input saturation.

The authors suggest several avenues for future research: extending the framework to multi‑degree‑of‑freedom mechanical systems, robotic manipulators, and power‑grid models; investigating scalability of the parameterization for high‑dimensional PH systems; and integrating formal safety verification (e.g., synthesis of Lyapunov certificates) with the learned policies. Overall, the paper provides a compelling bridge between classical energy‑shaping control theory and modern data‑driven reinforcement learning, opening the door to systematic, performance‑driven control synthesis for a broad class of physically grounded systems.