Scalable Multi-Objective Reinforcement Learning with Fairness Guarantees using Lorenz Dominance

Multi-Objective Reinforcement Learning (MORL) aims to learn a set of policies that optimize trade-offs between multiple, often conflicting objectives. MORL is computationally more complex than single-objective RL, particularly as the number of objectives increases. Additionally, when objectives involve the preferences of agents or groups, incorporating fairness becomes both important and socially desirable. This paper introduces a principled algorithm that incorporates fairness into MORL while improving scalability to many-objective problems. We propose using Lorenz dominance to identify policies with equitable reward distributions and introduce lambda-Lorenz dominance to enable flexible fairness preferences. We release a new, large-scale real-world transport planning environment and demonstrate that our method encourages the discovery of fair policies, showing improved scalability in two large cities (Xi’an and Amsterdam). Our methods outperform common multi-objective approaches, particularly in high-dimensional objective spaces.

💡 Research Summary

The paper tackles two intertwined challenges in Multi‑Objective Reinforcement Learning (MORL): guaranteeing fairness among objectives that often represent different societal groups, and scaling to many‑objective problems where the Pareto front becomes prohibitively large. Traditional MORL methods approximate the entire Pareto front, which grows exponentially with the number of objectives and includes many policies that are undesirable from a fairness perspective. Existing fairness‑aware RL approaches typically embed a specific fairness notion into the reward function (e.g., weighted sums, max‑min, Generalized Gini Index) and therefore require prior knowledge of stakeholder preferences.

To overcome these limitations, the authors introduce Lorenz dominance—a concept from economics that compares vectors after sorting their components in ascending order and then cumulatively summing them. A vector Lorenz‑dominates another if its cumulative sums are never lower, embodying the Pigou‑Dalton transfer principle: moving a small amount of reward from a better‑off component to a worse‑off component improves fairness without changing the total reward. Because the Lorenz front is a subset of the Pareto front, focusing on Lorenz‑optimal policies reduces the size of the solution set while providing strong equity guarantees.

The core methodological contribution is λ‑Lorenz dominance, a parameterised dominance relation that interpolates between pure Lorenz (λ = 1) and pure Pareto (λ = 0). By selecting λ, decision‑makers can control the strictness of fairness constraints after training, without having to specify any fairness weighting beforehand. Building on this, the authors propose Lorenz Conditioned Networks (LCN), a neural‑network architecture that learns a conditional policy mapping from a desired λ value to a set of policies that are non‑λ‑Lorenz‑dominated. Unlike Pareto Conditioned Networks (PCN), which aim to approximate the full Pareto front, LCN only needs to approximate the much smaller Lorenz front, dramatically lowering computational and memory demands.

The experimental evaluation uses two large‑scale, real‑world transport‑network design environments—Xi’an (China) and Amsterdam (Netherlands). Each environment is modelled as a Multi‑Objective Markov Decision Process (MOMDP) with 6 to 10 objectives (e.g., traffic flow, emissions, cost, travel time, resident satisfaction). Baselines include state‑of‑the‑the‑art multi‑policy MORL methods: GPI‑LS, IPRO, PCN, and C‑MORL. The authors assess (i) the size of the discovered policy set, (ii) training time and memory consumption, (iii) overall utility (sum of objectives), and (iv) fairness metrics such as the Gini coefficient and the area under the Lorenz curve.

Results show that LCN consistently produces a policy set that is 40‑60 % smaller than the Pareto front while preserving comparable total utility. As λ increases, the Gini coefficient drops substantially, confirming that the returned policies are more equitable. In high‑dimensional settings (≥ 8 objectives), traditional baselines either fail to converge or generate an explosion of policies, whereas LCN remains stable and yields only a few dozen policies. The authors also provide theoretical analysis proving that λ‑Lorenz dominance defines a partial order that smoothly transitions between Pareto and Lorenz dominance, and that the Lorenz front’s subset property guarantees that any Lorenz‑optimal policy is also Pareto‑optimal.

Limitations are acknowledged: (1) Lorenz dominance captures only one notion of fairness and may not satisfy other equity criteria such as minimum‑guarantee or dynamic preference changes; (2) selecting λ currently requires post‑hoc analysis, suggesting a need for automated λ‑tuning mechanisms; (3) experiments are confined to static transport scenarios, leaving real‑time adaptation (e.g., accidents, demand spikes) for future work. The authors propose extending the framework to incorporate alternative inequality measures (Atkinson, Theil) and to handle multi‑agent settings where multiple groups’ Lorenz fronts must be balanced simultaneously.

In summary, the paper presents a principled, scalable approach to fairness‑aware MORL by leveraging Lorenz dominance and introducing λ‑Lorenz dominance together with Lorenz Conditioned Networks. It demonstrates that equitable policy sets can be learned efficiently even in many‑objective, real‑world environments, offering a significant advance over existing MORL and fairness‑focused RL methods.