Deep Pareto Reinforcement Learning for Multi-Objective Recommender Systems

Optimizing multiple objectives simultaneously is an important task for recommendation platforms to improve their performance. However, this task is particularly challenging since the relationships between different objectives are heterogeneous across different consumers and dynamically fluctuating according to different contexts. Especially in those cases when objectives become conflicting with each other, the result of recommendations will form a pareto-frontier, where the improvements of any objective comes at the cost of a performance decrease of another objective. Existing multi-objective recommender systems do not systematically consider such dynamic relationships; instead, they balance between these objectives in a static and uniform manner, resulting in only suboptimal multi-objective recommendation performance. In this paper, we propose a Deep Pareto Reinforcement Learning (DeepPRL) approach, where we (1) comprehensively model the complex relationships between multiple objectives in recommendations; (2) effectively capture personalized and contextual consumer preference for each objective to provide better recommendations; (3) optimize both the short-term and the long-term performance of multi-objective recommendations. As a result, our method achieves significant pareto-dominance over the state-of-the-art baselines in the offline experiments. Furthermore, we conducted a controlled experiment at the video streaming platform of Alibaba, where our method simultaneously improved three conflicting business objectives over the latest production system significantly, demonstrating its tangible economic impact in practice.

💡 Research Summary

The paper tackles the challenging problem of simultaneously optimizing multiple, often conflicting, objectives in recommender systems—such as click‑through rate (CTR), video view count (VV), and dwell time (DT). Existing multi‑objective recommender approaches typically rely on static, globally fixed weights or treat each objective independently, ignoring the fact that the relationships among objectives vary across users and contexts (time of day, device, recent activity, etc.). This leads to sub‑optimal performance and an inability to move toward the true Pareto frontier where no objective can be improved without degrading another.

To address these gaps, the authors propose Deep Pareto Reinforcement Learning (DeepPRL), a unified framework that (1) explicitly models the heterogeneous, dynamic relationships among objectives, (2) captures personalized and contextual preferences for each objective, and (3) optimizes both short‑term and long‑term performance through reinforcement learning.

DeepPRL consists of two novel components. The “Mixture of HyperNetwork” builds a separate hypernetwork for each objective; each hypernetwork generates objective‑specific parameters conditioned on user, item, and contextual embeddings. The resulting parameter sets are then fused by a mixture‑attention network that learns how the objectives interact (positively or negatively) for a given user‑context pair. This design enables the system to adaptively select products that improve the Pareto frontier with respect to the current trade‑off landscape.

The second component, “Deep Contextual Reinforcement Learning,” treats recommendation as a Markov Decision Process with a multi‑dimensional reward vector (one dimension per objective). A context‑aware weight generator network produces dynamic objective weights wₖ(c) in real time, allowing the policy to balance immediate rewards (e.g., CTR) against future rewards (e.g., dwell time, repeat visits). The policy is trained with a Pareto‑gradient actor‑critic algorithm that explicitly seeks improvements on the Pareto front, and the authors provide theoretical proofs of convergence and Pareto dominance guarantees.

Empirically, the authors evaluate DeepPRL on four large‑scale datasets (Alibaba‑Youku, Yelp, Spotify, Kuaishou) and compare against strong baselines such as Shared‑Bottom, Multi‑Gate Mixture‑of‑Experts (MMOE), MoSE, and recent Pareto‑oriented methods. Across all datasets, DeepPRL achieves substantial gains in Pareto metrics (hyper‑volume, domination ratio) and standard recommendation metrics (NDCG, CTR). A simulation study further demonstrates that the learned policy maintains higher long‑term rewards than static‑weight baselines.

Crucially, a live A/B test on Alibaba’s video streaming platform shows that deploying DeepPRL improves CTR by 2 %, VV by 5 %, and DT by 7 % simultaneously, translating into measurable revenue and user‑engagement uplift.

The paper’s contributions are: (1) a principled way to model and learn heterogeneous, dynamic objective relationships; (2) a context‑driven, real‑time weight adjustment mechanism that bridges short‑ and long‑term goals; (3) theoretical guarantees of Pareto optimality; and (4) validation of both offline and online performance at industrial scale. Limitations include increased model complexity and potential scalability issues when the number of objectives grows dramatically. Future work is suggested on graph‑based objective interaction modeling, meta‑learning for faster context adaptation, and scalable Pareto optimization for many‑objective settings.