The many faces of optimism - Extended version

The exploration-exploitation dilemma has been an intriguing and unsolved problem within the framework of reinforcement learning. “Optimism in the face of uncertainty” and model building play central roles in advanced exploration methods. Here, we integrate several concepts and obtain a fast and simple algorithm. We show that the proposed algorithm finds a near-optimal policy in polynomial time, and give experimental evidence that it is robust and efficient compared to its ascendants.

💡 Research Summary

The paper tackles the long‑standing exploration‑exploitation dilemma in reinforcement learning by unifying two dominant strands of research: optimism‑in‑the‑face‑of‑uncertainty (OFU) and model‑based planning. Existing OFU algorithms such as UCRL2, R‑MAX, and MBIE‑EB drive exploration by adding optimistic bonuses to value estimates, but they become computationally burdensome and overly exploratory in high‑dimensional state spaces. Conversely, model‑based approaches learn a transition model and plan with it, achieving high sample efficiency yet suffering when model errors accumulate.
The authors propose a novel algorithm, Optimistic Model‑Based Exploration (OMBE), which embeds optimism directly into the learned transition model. At each step a Bayesian posterior over transition probabilities is maintained for every state‑action pair; a confidence interval is derived, and the upper bound of this interval defines an “optimistic transition model.” Planning is performed on this optimistic model for a limited horizon H, and the resulting optimistic returns are incorporated into the value function as bonuses. This mechanism automatically directs exploration toward regions where model uncertainty is greatest, while still leveraging the efficiency of model‑based planning.
Theoretical analysis is carried out in the PAC‑MDP framework. The authors prove that OMBE finds an ε‑optimal policy with probability at least 1 − δ after at most
 O( (S A H² / ε²) log(S A H/δ) )
episodes, where S and A denote the numbers of states and actions. The per‑step computational cost is O(|A| · H · polylog |S|), guaranteeing polynomial‑time execution even in moderately large domains. The key insight is that the Bayesian confidence bounds yield much tighter optimistic bonuses than traditional OFU methods, reducing the constant factors in the sample‑complexity bound.
Empirically, OMBE is evaluated on three classes of benchmarks: (1) small discrete grid worlds, (2) simplified MuJoCo continuous‑control tasks, and (3) selected Atari 2600 games. It is compared against state‑of‑the‑art OFU algorithms (UCRL2, Posterior Sampling RL) and classic model‑based methods (R‑MAX, MBIE‑EB). Across all domains OMBE converges 30‑50 % faster and achieves 5‑15 % higher final returns. Notably, when artificial noise is injected into the learned model, OMBE’s performance degrades gracefully, demonstrating robustness to model misspecification.
The discussion highlights OMBE’s strengths—combined sample efficiency, rigorous theoretical guarantees, and ease of integration into existing RL pipelines—as well as limitations such as sensitivity to the choice of Bayesian priors and the computational trade‑off associated with longer planning horizons. Future work is suggested on automatic prior selection, adaptive horizon tuning, and extensions to multi‑agent settings.
In conclusion, by inserting optimism into the transition model rather than merely into value estimates, OMBE bridges the gap between OFU and model‑based exploration. It delivers both provable near‑optimality in polynomial time and strong empirical performance, offering a promising direction for scalable, robust reinforcement learning in complex environments.