The Non-Bayesian Restless Multi-Armed Bandit: A Case of Near-Logarithmic Strict Regret

In the classic Bayesian restless multi-armed bandit (RMAB) problem, there are $N$ arms, with rewards on all arms evolving at each time as Markov chains with known parameters. A player seeks to activate $K \geq 1$ arms at each time in order to maximize the expected total reward obtained over multiple plays. RMAB is a challenging problem that is known to be PSPACE-hard in general. We consider in this work the even harder non-Bayesian RMAB, in which the parameters of the Markov chain are assumed to be unknown \emph{a priori}. We develop an original approach to this problem that is applicable when the corresponding Bayesian problem has the structure that, depending on the known parameter values, the optimal solution is one of a prescribed finite set of policies. In such settings, we propose to learn the optimal policy for the non-Bayesian RMAB by employing a suitable meta-policy which treats each policy from this finite set as an arm in a different non-Bayesian multi-armed bandit problem for which a single-arm selection policy is optimal. We demonstrate this approach by developing a novel sensing policy for opportunistic spectrum access over unknown dynamic channels. We prove that our policy achieves near-logarithmic regret (the difference in expected reward compared to a model-aware genie), which leads to the same average reward that can be achieved by the optimal policy under a known model. This is the first such result in the literature for a non-Bayesian RMAB. For our proof, we also develop a novel generalization of the Chernoff-Hoeffding bound.

💡 Research Summary

The paper tackles the non‑Bayesian Restless Multi‑Armed Bandit (RMAB) problem, where the transition probabilities of the underlying Markov chains are unknown to the decision maker. While RMAB is already PSPACE‑hard even when the parameters are known, the authors focus on a subclass of Bayesian RMAB problems that admit a finite‑option structure: the parameter space can be partitioned into a finite number of regions, and for each region a single deterministic policy is provably optimal. They denote this class as Ψₘ.

The key insight is to treat each of these optimal‑for‑a‑region policies as an arm in a separate non‑Bayesian multi‑armed bandit problem. In this meta‑bandit, the goal is to identify which policy yields the highest expected reward under the true (but unknown) parameters. Classical exploration‑exploitation algorithms such as Lai‑Robbins or UCB can be applied to this meta‑problem. However, a practical difficulty arises: each “arm” (i.e., each candidate policy) must be executed for a certain number of time steps before its performance can be assessed, and the appropriate horizon depends on the unknown parameters. To overcome this, the authors propose a slowly increasing execution length: initially each policy is run for a short block, and the block size grows over time, guaranteeing enough samples for reliable estimation while keeping early regret low.

The framework is instantiated on the opportunistic spectrum access scenario. A secondary user observes N independent two‑state Markov channels, each with the same unknown transition matrix P. At each slot the user can sense one channel; if the sensed state is “1” (idle) a unit reward is obtained. Prior work has shown that, depending on the sign of the correlation (p₁₁ ≥ p₀₁ or p₁₁ < p₀₁), one of two myopic policies (π₁ or π₂) is optimal when P is known. Hence the problem belongs to Ψ₂. The meta‑policy treats π₁ and π₂ as two arms and learns which one matches the true P.

A novel contribution is a generalized Chernoff‑Hoeffding bound that accommodates the dependence structure of Markovian rewards. Using this bound, the authors prove that for N = 2 and N = 3 the cumulative regret R(n) of their meta‑policy satisfies

 R(n) = O(G(n)·log n),

where G(n) is any arbitrarily slowly diverging non‑decreasing function (effectively a constant in practice). Thus the regret grows only logarithmically up to a negligible factor, which is essentially the best possible rate for bandit problems. For general N, extensive simulations suggest that the myopic policy remains optimal, and the meta‑policy achieves the same average reward as the genie that knows P.

Experimental results confirm: (1) rapid discrimination between the two candidate policies, (2) average reward indistinguishable from the optimal Bayesian policy, and (3) superior performance compared with naïve non‑Bayesian RMAB approaches (e.g., applying UCB directly to each channel). The increasing‑block schedule also provides robustness to slowly varying channel statistics.

In summary, the paper introduces a principled method for non‑Bayesian RMABs when the Bayesian counterpart exhibits a finite‑policy partition. By converting the problem into a meta‑bandit over policies and carefully scheduling policy execution, it achieves near‑logarithmic strict regret—first of its kind for non‑Bayesian RMABs. The generalized concentration inequality further enriches the theoretical toolbox for analyzing Markov‑dependent reward processes. This work bridges the gap between Bayesian optimality results and practical learning algorithms in restless environments, with immediate implications for dynamic spectrum access and other applications where system dynamics are unknown.