Maximum Entropy Exploration Without the Rollouts

Efficient exploration remains a central challenge in reinforcement learning, serving as a useful pretraining objective for data collection, particularly when an external reward function is unavailable. A principled formulation of the exploration problem is to find policies that maximize the entropy of their induced steady-state visitation distribution, thereby encouraging uniform long-run coverage of the state space. Many existing exploration approaches require estimating state visitation frequencies through repeated on-policy rollouts, which can be computationally expensive. In this work, we instead consider an intrinsic average-reward formulation in which the reward is derived from the visitation distribution itself, so that the optimal policy maximizes steady-state entropy. An entropy-regularized version of this objective admits a spectral characterization: the relevant stationary distributions can be computed from the dominant eigenvectors of a problem-dependent transition matrix. This insight leads to a novel algorithm for solving the maximum entropy exploration problem, EVE (EigenVector-based Exploration), which avoids explicit rollouts and distribution estimation, instead computing the solution through iterative updates, similar to a value-based approach. To address the original unregularized objective, we employ a posterior-policy iteration (PPI) approach, which monotonically improves the entropy and converges in value. We prove convergence of EVE under standard assumptions and demonstrate empirically that it efficiently produces policies with high steady-state entropy, achieving competitive exploration performance relative to rollout-based baselines in deterministic grid-world environments.

💡 Research Summary

The paper tackles the problem of exploration in reinforcement learning when no external reward signal is available. The authors formalize exploration as the maximization of the entropy of the stationary state‑action visitation distribution induced by a policy, thereby encouraging uniform long‑run coverage of the environment. Traditional approaches estimate this distribution through repeated on‑policy rollouts, which is computationally costly because each policy update requires fresh visitation statistics.

To avoid rollouts, the authors recast the problem in an average‑reward framework. They define an intrinsic reward (r(s,a) = -\log d_{p,\pi}(s,a)), where (d_{p,\pi}) is the stationary distribution under transition dynamics (p) and policy (\pi). Maximizing the average reward with this intrinsic reward is equivalent to maximizing the entropy of (d_{p,\pi}). They further introduce an entropy‑regularized version of the objective, adding a KL‑penalty to a reference policy (\pi_0) (typically uniform) with inverse temperature (\beta).

A key theoretical contribution is the construction of a “tilted matrix”
\