Preference elicitation and inverse reinforcement learning

We state the problem of inverse reinforcement learning in terms of preference elicitation, resulting in a principled (Bayesian) statistical formulation. This generalises previous work on Bayesian inverse reinforcement learning and allows us to obtain a posterior distribution on the agent’s preferences, policy and optionally, the obtained reward sequence, from observations. We examine the relation of the resulting approach to other statistical methods for inverse reinforcement learning via analysis and experimental results. We show that preferences can be determined accurately, even if the observed agent’s policy is sub-optimal with respect to its own preferences. In that case, significantly improved policies with respect to the agent’s preferences are obtained, compared to both other methods and to the performance of the demonstrated policy.

💡 Research Summary

The paper “Preference Elicitation and Inverse Reinforcement Learning” reframes the inverse reinforcement learning (IRL) problem as a preference‑elicitation task and provides a principled Bayesian statistical treatment. The authors consider an agent operating in a known controlled Markov process (S, A, T) with an unknown stochastic reward function ρ(s,a) and discount factor γ. The agent’s utility at time t is the discounted sum of future rewards, Uₜ = ∑_{k=t}^∞ γ^{k‑t} r_k, where each r_k is drawn from ρ conditioned on the current state‑action pair. The agent follows a policy π that is assumed to be “nearly optimal” with respect to its own utility, but not necessarily exactly optimal.

Two probabilistic graphical models are introduced. The basic model treats the reward function ρ and the policy π as latent variables with priors ξ(· | ν) over rewards and ψ(· | ρ, ν) over policies, where ν denotes the known environment dynamics. Observations consist of a trajectory D = (s₁:T, a₁:T). The posterior over rewards is proportional to the marginal likelihood of the observed actions under the policy, integrated over the policy prior. The second, “reward‑augmented” model additionally makes the actual reward sequence r₁:T explicit, allowing a joint posterior over (ρ, π, r₁:T).

For inference, the authors propose two sampling schemes. In the basic model, a Metropolis‑Hastings (MH) algorithm samples jointly from (ρ, π). The policy is restricted to a soft‑max form π_η(a|s) = exp(η Q⁎μ(s,a))/∑{a’}exp(η Q⁎_μ(s,a’)), where Q⁎_μ is the optimal Q‑function for the MDP defined by (ρ, γ) and η (inverse temperature) has a Gamma prior. At each MH iteration a new reward function ρ̃ is drawn from the reward prior, a new η̃ from the Gamma prior, the corresponding soft‑max policy π̃ is computed, and the acceptance probability is based on the likelihood of the observed trajectory under π̃.

The reward‑augmented model uses a hybrid Gibbs‑MH sampler. Given the current reward function and η, a new policy is proposed as above (MH step). Then, conditioned on the proposed ρ and the observed state‑action pairs, a new reward sequence r₁:T is sampled directly from the reward distribution. This two‑stage procedure yields a posterior not only on ρ and π but also on the latent rewards, which can be valuable in applications where the reward trajectory itself is of interest.

Theoretical analysis shows that the transition probabilities τ(s′|s,a) cancel out of the posterior, leaving the likelihood dependent only on the policy’s action probabilities. Consequently, even when the observed policy deviates from optimality, the Bayesian framework can still recover the underlying reward function provided the priors are suitably expressive.

Empirical evaluation is performed on grid‑world style MDPs. The proposed method is compared against Maximum Entropy IRL, earlier Bayesian IRL approaches, and linear‑reward IRL. Metrics include (i) mean‑squared error between the inferred and true reward functions, (ii) expected utility of the policy derived from the inferred reward, and (iii) improvement over the demonstrator’s original policy measured with respect to the true utility. Results demonstrate that the Bayesian preference‑elicitation approach achieves lower reward reconstruction error and yields policies that significantly outperform the demonstrator’s policy, even when the demonstrator’s behavior is sub‑optimal relative to its own preferences.

Key contributions are: (1) a unified Bayesian formulation of IRL as preference elicitation, providing full posterior distributions over rewards, policies, and optionally reward sequences; (2) relaxation of the strict optimal‑policy assumption by employing a soft‑max policy with an inferred temperature parameter, enabling robust inference under near‑optimal behavior; (3) an augmented model that jointly infers latent reward trajectories, opening avenues for domains where reward signals are of primary interest (e.g., behavioral economics, human‑robot interaction); and (4) practical sampling algorithms (MH and hybrid Gibbs‑MH) that are simple to implement yet empirically effective.

The paper concludes with suggestions for future work: extending the policy class beyond soft‑max, integrating active experimental design to query the agent for informative demonstrations, and scaling the inference to high‑dimensional continuous state‑action spaces using variational or particle‑based approximations. Overall, the work bridges the gap between decision‑theoretic preference elicitation and inverse reinforcement learning, offering a statistically sound and practically useful framework for learning hidden preferences from observed behavior.