A Unifying Framework for Linearly Solvable Control

Recent work has led to the development of an elegant theory of Linearly Solvable Markov Decision Processes (LMDPs) and related Path-Integral Control Problems. Traditionally, MDPs have been formulated using stochastic policies and a control cost based on the KL divergence. In this paper, we extend this framework to a more general class of divergences: the Renyi divergences. These are a more general class of divergences parameterized by a continuous parameter that include the KL divergence as a special case. The resulting control problems can be interpreted as solving a risk-sensitive version of the LMDP problem. For a > 0, we get risk-averse behavior (the degree of risk-aversion increases with a) and for a < 0, we get risk-seeking behavior. We recover LMDPs in the limit as a -> 0. This work generalizes the recently developed risk-sensitive path-integral control formalism which can be seen as the continuous-time limit of results obtained in this paper. To the best of our knowledge, this is a general theory of linearly solvable control and includes all previous work as a special case. We also present an alternative interpretation of these results as solving a 2-player (cooperative or competitive) Markov Game. From the linearity follow a number of nice properties including compositionality of control laws and a path-integral representation of the value function. We demonstrate the usefulness of the framework on control problems with noise where different values of lead to qualitatively different control behaviors.

💡 Research Summary

The paper presents a unifying theory that extends the class of linearly solvable Markov decision processes (LMDPs) by replacing the traditional Kullback‑Leibler (KL) divergence‑based control cost with the more general Rényi divergence. Rényi divergence Dα(p‖q)= (1/(α‑1)) log ∫p(x)^α q(x)^{1‑α}dx is parameterized by a real scalar α; when α→0 it reduces to the KL divergence, and the corresponding LMDP formulation is recovered. By embedding Dα into the instantaneous cost, the authors obtain a risk‑sensitive control problem: for α>0 the controller behaves risk‑averse, for α<0 it becomes risk‑seeking, and the degree of risk sensitivity scales with |α|.

Through a variational derivation, the optimal value function V(s) is transformed into a “partition” function Z(s)=exp(‑V(s)/α). The Bellman equation becomes a linear system:
 Z(s)=∑_{s’} P(s’|s) exp(‑c(s,s’)/α) Z(s’).
Thus, despite the non‑linear appearance of the original problem, the transformed equations are linear in Z for any α, preserving the hallmark property of LMDPs. This linearity yields several powerful consequences:

1. Compositionality – solutions for multiple sub‑goals can be linearly combined to form a solution for a composite objective, enabling modular policy synthesis.
2. Path‑Integral Representation – Z(s) can be expressed as a weighted sum over all stochastic trajectories, allowing Monte‑Carlo sampling to approximate the value function even in high‑dimensional continuous spaces.
3. Continuous‑Time Limit – taking the time‑step Δt→0 recovers the previously known risk‑sensitive path‑integral control equations, showing that the discrete‑time formulation is a strict generalization.

The authors also reinterpret the framework as a two‑player Markov game. One player represents the controller, the other the environment (or an adversary). In a cooperative setting both players minimize the same cost; in a competitive setting one maximizes it. The Rényi parameter α acts as a “cooperation‑competition knob,” smoothly interpolating between fully cooperative (α=0) and strongly competitive (large |α|) regimes.

Empirical validation is provided on noisy nonlinear systems such as a double pendulum and a 6‑DOF robotic arm. Varying α demonstrates qualitatively different behaviors: risk‑averse policies (α>0) follow smoother, more conservative trajectories and tolerate higher noise at the expense of slower task completion; risk‑seeking policies (α<0) pursue aggressive, faster paths but are more prone to failure under disturbances. The KL‑based LMDP (α≈0) sits between these extremes, confirming the theoretical predictions.

In summary, the paper establishes a comprehensive, mathematically elegant extension of linearly solvable control. By leveraging Rényi divergences, it unifies risk‑neutral LMDPs, risk‑sensitive path‑integral control, and a game‑theoretic perspective under a single linear framework. The resulting properties—linearity, compositionality, and a clear risk‑sensitivity parameter—make the approach attractive for a broad spectrum of applications, from robotics and autonomous navigation to finance and multi‑agent systems. Future directions suggested include state‑dependent α adaptation, learning the base dynamics μ from data, and extending the game formulation to many‑agent competitive environments.