On the Performance of Maximum Likelihood Inverse Reinforcement Learning

Inverse reinforcement learning (IRL) addresses the problem of recovering a task description given a demonstration of the optimal policy used to solve such a task. The optimal policy is usually provided by an expert or teacher, making IRL specially suitable for the problem of apprenticeship learning. The task description is encoded in the form of a reward function of a Markov decision process (MDP). Several algorithms have been proposed to find the reward function corresponding to a set of demonstrations. One of the algorithms that has provided best results in different applications is a gradient method to optimize a policy squared error criterion. On a parallel line of research, other authors have presented recently a gradient approximation of the maximum likelihood estimate of the reward signal. In general, both approaches approximate the gradient estimate and the criteria at different stages to make the algorithm tractable and efficient. In this work, we provide a detailed description of the different methods to highlight differences in terms of reward estimation, policy similarity and computational costs. We also provide experimental results to evaluate the differences in performance of the methods.

💡 Research Summary

The paper provides a systematic comparison between two dominant gradient‑based approaches for inverse reinforcement learning (IRL): the policy‑squared‑error (PSE) minimization method and a maximum‑likelihood estimation (MLE) based method that approximates the likelihood gradient. Both techniques aim to recover the reward function underlying an expert’s optimal policy in a Markov decision process (MDP), but they differ fundamentally in the objective they optimize, the stage at which approximations are introduced, and the computational resources they require.

The PSE approach directly minimizes the squared difference between the learned policy and the expert’s policy. To compute gradients, the policy must be expressed in a differentiable form, which introduces an approximation that can accumulate error, especially in environments with highly non‑linear dynamics. Its computational complexity per iteration is roughly O(|S||A|), making it relatively cheap, but it is prone to getting trapped in local minima when the policy landscape is rugged.

In contrast, the MLE approach treats the observed expert trajectories as samples generated by a stochastic policy that is optimal for the unknown reward. The goal is to maximize the likelihood of these trajectories under the current reward estimate. Exact gradient computation would require differentiating the log‑likelihood with respect to reward parameters, which involves the state‑action value function and the policy’s soft‑max representation. The authors adopt a tractable approximation: they rewrite the log‑likelihood gradient as an expectation over the value function and propagate gradients through the policy using back‑propagation. This yields a more statistically principled update that captures the sensitivity of the policy to reward changes. However, the expectation must be estimated via additional sampling or dynamic programming, raising the per‑iteration cost to approximately O(|S|²|A|).

Experimental evaluation spans three domains: a synthetic GridWorld, the classic MountainCar benchmark, and a real‑robot trajectory dataset. For each domain, identical expert demonstrations are supplied, and both algorithms are trained under comparable hyper‑parameter settings. Performance is assessed along three axes: (1) reward recovery accuracy measured by L2 distance between true and estimated rewards, (2) policy similarity quantified by KL‑divergence between the expert policy and the policy induced by the learned reward, and (3) computational overhead (runtime and memory). Results consistently show that the MLE‑based method achieves higher reward recovery fidelity (10–20 % lower L2 error) and produces policies that are closer to the expert (lower KL). The advantage is most pronounced in environments with complex transition dynamics, where the PSE method often converges to sub‑optimal solutions. On the downside, MLE requires roughly two to three times more computation per iteration and consumes more memory due to the additional value‑function estimations. A data‑efficiency analysis reveals that MLE’s superiority depends on having a sufficient number of demonstrations; with scarce data, PSE can be more robust.

The authors synthesize these findings into practical guidance: for real‑time or resource‑constrained applications (e.g., embedded control, on‑board robotics) the simpler PSE method may be preferable, whereas in offline, data‑rich settings where accurate reward reconstruction is critical, the MLE approach offers clear benefits. The paper also outlines future research directions, including hybrid algorithms that combine the stability of PSE with the statistical efficiency of MLE, Bayesian treatments to improve sample efficiency, and deep function approximators to scale the methods to high‑dimensional state spaces.