Interpretability by Design for Efficient Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning (MORL) aims at optimising several, often conflicting goals to improve the flexibility and reliability of RL in practical tasks. This is typically achieved by finding a set of diverse, non-dominated policies that form a Pareto front in the performance space. We introduce LLE-MORL, an approach that achieves interpretability by design by utilising a training scheme based on the local relationship between the parameter space and the performance space. By exploiting a locally linear map between these spaces, our method provides an interpretation of policy parameters in terms of the objectives, and this structured representation enables an efficient search within contiguous solution domains, allowing for the rapid generation of high-quality solutions without extensive retraining. Experiments across diverse continuous control domains demonstrate that LLE-MORL consistently achieves higher Pareto front quality and efficiency than state-of-the-art approaches.

💡 Research Summary

The paper tackles three intertwined challenges in multi‑objective reinforcement learning (MORL): (1) the high sample cost of learning a set of policies that cover the Pareto front, (2) the difficulty of representing these policies in a way that allows smooth adaptation to changing user preferences, and (3) the lack of interpretability that makes it hard for practitioners to understand how changes in policy parameters affect the trade‑offs among objectives.
The authors hypothesize that, although a global one‑to‑one mapping between policy parameters and multi‑objective performance rarely exists, local linear relationships do exist in many practical problems. They formalize this as the Parameter‑Performance Relationship (PPR): within a small neighbourhood of the parameter space, a perturbation Δθ leads to a predictable change in the expected return vector V(θ) that can be expressed by a linear function h(θ,Δθ).
To validate PPR, they train two policies on the same environment with different scalarisation weights ω₁ and ω₂, then perform a single short retraining step on the first policy using ω₂. The resulting policy θ′ is shown (via Hungarian matching distance and heat‑map visualisations) to be structurally close to the original θ₁ while its performance moves in the direction dictated by the new preference. This demonstrates that brief retraining produces locally related policies that satisfy PPR.
Building on this observation, the authors introduce Locally Linear Extrapolation (LLE). Given a base policy θ and a PPR‑compatible retrained policy θ′, the difference Δθ = θ′ – θ defines a direction in parameter space that corresponds to a specific trade‑off between objectives. By scaling this direction with factors α drawn from a predefined grid, they generate a family of intermediate policies θ_α = θ + αΔθ. Each θ_α is evaluated in the multi‑objective space; as α varies, the resulting performance points trace a smooth curve that passes through both the base and retrained policies and can extend beyond them, effectively exploring a continuous segment of the Pareto front without training each point from scratch.
The full LLE‑MORL algorithm consists of five stages:

1. Initialisation – train K base policies with PPO under evenly spaced preference vectors ω_k.
2. Directional Retraining – for each base policy, perform short PPO retraining under d‑1 nearby preferences to obtain directional policies and record both parameter updates Δθ_(i) and preference shifts Δω_(i).
3. Locally Linear Extension – combine the recorded directions linearly with scale factors α_i to generate M^m candidate policies per base policy, where m = d‑1. Each candidate inherits a matched preference vector constructed from the same α_i weights.
4. Candidate Selection – evaluate all candidates, keep only the non‑dominated ones, and discard the rest.
5. Preference‑Aligned Fine‑Tuning – apply a brief PPO fine‑tuning phase (T_ref steps) to each selected candidate under its matched preference, nudging it toward the true Pareto front.
   Theoretical analysis (Appendix B) models the Pareto front as a d‑dimensional manifold embedded in the high‑dimensional parameter space. Under Lipschitz continuity assumptions, the authors prove that linear extrapolation along locally linear directions preserves non‑domination with high probability, providing a formal guarantee for the LLE step.
   Empirical evaluation on several MuJoCo continuous‑control benchmarks (e.g., SWIMMER, HalfCheetah‑Multi, Walker2D‑Multi) shows that LLE‑MORL consistently outperforms state‑of‑the‑art MORL methods such as PGMORL, MORL/D, and Pareto‑CMA‑ES. Metrics include hypervolume (improved by 5–15 %), sample efficiency (30–50 % fewer training steps for comparable hypervolume), and interpretability (visual heat‑maps and Hungarian distances that link parameter changes to objective trade‑offs).
   In summary, LLE‑MORL introduces a design‑time interpretability by embedding a locally linear parameter‑performance map into the learning process, and leverages this structure for sample‑efficient Pareto front exploration. The approach reduces the need for exhaustive retraining, provides a clear semantic link between policy parameters and objective trade‑offs, and delivers high‑quality Pareto approximations, making it a compelling advancement for practical multi‑objective reinforcement learning applications.