Rollout Sampling Approximate Policy Iteration

Several researchers have recently investigated the connection between reinforcement learning and classification. We are motivated by proposals of approximate policy iteration schemes without value functions which focus on policy representation using classifiers and address policy learning as a supervised learning problem. This paper proposes variants of an improved policy iteration scheme which addresses the core sampling problem in evaluating a policy through simulation as a multi-armed bandit machine. The resulting algorithm offers comparable performance to the previous algorithm achieved, however, with significantly less computational effort. An order of magnitude improvement is demonstrated experimentally in two standard reinforcement learning domains: inverted pendulum and mountain-car.

💡 Research Summary

The paper addresses a major inefficiency in Rollout Classification Policy Iteration (RCPI), a method that learns policies directly as classifiers by generating training data through Monte‑Carlo rollouts. In standard RCPI each state in a sampled set S_R receives a fixed budget of K rollouts per action, regardless of how informative those rollouts are. Consequently, many rollouts are wasted on states where the best action is already obvious or where all actions have indistinguishable values.

The authors recast the rollout‑allocation problem as a multi‑armed bandit (MAB) task. Each state corresponds to an arm; pulling an arm means performing one rollout for every action in that state (the minimal information unit). By using bandit strategies they can allocate rollouts adaptively, stopping early on states whose dominant action can be identified with high confidence, and discarding states where actions are statistically indistinguishable.

Three bandit‑based policies are investigated:

1. Count – simply selects the least‑sampled state, ensuring uniform coverage.
2. UCB1 / UCB2 – combines an empirical estimate of the action‑value gap Δ̂π(s) with an exploration bonus (1/(1+c(s)) or ln m/(1+c(s))) to prioritize states that appear promising yet under‑explored.
3. Successive Elimination – after each rollout batch, applies a statistical test (e.g., confidence intervals or t‑test) to eliminate arms whose dominant action is already determined, or to reject arms where no action dominates.

The algorithm proceeds as follows: a pool S_R of states is initially drawn uniformly at random. At each iteration the state s maximizing a utility function U(s) (depending on the chosen bandit rule) is selected. A single “sample” of s consists of one rollout for every action, after which Q̂π(s,a) and the empirical gap Δ̂π(s) are updated. If Δ̂π(s) exceeds a pre‑set threshold or a statistical test confirms a dominant action, the corresponding (state, action) label(s) are added to the training set and s is removed from S_R. If all actions appear equal, s is discarded and a fresh state is inserted. The process repeats until the policy’s performance (measured by simulation) no longer improves.

Empirical evaluation on two classic reinforcement‑learning benchmarks – the inverted pendulum and the mountain‑car – demonstrates that the bandit‑augmented RCPI achieves comparable or slightly better control performance while reducing the total number of rollouts by an order of magnitude. For example, where standard RCPI required 10–30 rollouts per state‑action pair, the UCB‑based version typically needed only 3–5, cutting simulation time dramatically.

Key contributions:

• Conceptual mapping of rollout allocation to a multi‑armed bandit framework, providing a principled way to balance exploration (sampling under‑explored states) and exploitation (focusing on states where a clear best action can be confirmed).
• Practical algorithms that integrate empirical action‑value gaps (Δ̂π) with classic bandit confidence bounds, and a successive‑elimination scheme that stops sampling as soon as statistical certainty is reached.
• Experimental validation showing that adaptive sampling does not degrade policy quality while delivering substantial computational savings, making RCPI viable for larger‑scale or real‑time applications.

The paper also discusses limitations. The current approach only manages sampling at the state level; extending bandit control to the action‑within‑state level could yield further gains. The reliability of Δ̂π depends on rollout horizon T and reward variance, so in highly stochastic domains the early‑stop criteria may be less robust. Finally, the uniform random initialization of S_R is simple but sub‑optimal; future work could bias state selection toward the γ‑discounted visitation distribution of the current policy or employ more sophisticated importance sampling.

Overall, the work presents a clear, theoretically grounded, and empirically effective method to dramatically reduce the sampling burden of classifier‑based policy iteration, opening the door to applying RCPI in more complex, real‑world reinforcement‑learning problems.