Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping

In this paper, we aim to tackle the limitation of the Adversarial Inverse Reinforcement Learning (AIRL) method in stochastic environments where theoretical results cannot hold and performance is degraded. To address this issue, we propose a novel method which infuses the dynamics information into the reward shaping with the theoretical guarantee for the induced optimal policy in the stochastic environments. Incorporating our novel model-enhanced rewards, we present a novel Model-Enhanced AIRL framework, which integrates transition model estimation directly into reward shaping. Furthermore, we provide a comprehensive theoretical analysis of the reward error bound and performance difference bound for our method. The experimental results in MuJoCo benchmarks show that our method can achieve superior performance in stochastic environments and competitive performance in deterministic environments, with significant improvement in sample efficiency, compared to existing baselines.

💡 Research Summary

The paper addresses a critical shortcoming of Adversarial Inverse Reinforcement Learning (AIRL) in stochastic Markov Decision Processes (MDPs), where the standard maximum‑entropy formulation fails to capture transition uncertainty, leading to degraded performance. To overcome this, the authors propose a model‑enhanced IRL framework that directly incorporates a learned transition model into reward shaping. Specifically, they define a transition‑aware reward ˆR(s,a,T̂)=R(s,a)+γ·E_T̂