Learning in A Changing World: Restless Multi-Armed Bandit with Unknown Dynamics

We consider the restless multi-armed bandit (RMAB) problem with unknown dynamics in which a player chooses M out of N arms to play at each time. The reward state of each arm transits according to an unknown Markovian rule when it is played and evolves according to an arbitrary unknown random process when it is passive. The performance of an arm selection policy is measured by regret, defined as the reward loss with respect to the case where the player knows which M arms are the most rewarding and always plays the M best arms. We construct a policy with an interleaving exploration and exploitation epoch structure that achieves a regret with logarithmic order when arbitrary (but nontrivial) bounds on certain system parameters are known. When no knowledge about the system is available, we show that the proposed policy achieves a regret arbitrarily close to the logarithmic order. We further extend the problem to a decentralized setting where multiple distributed players share the arms without information exchange. Under both an exogenous restless model and an endogenous restless model, we show that a decentralized extension of the proposed policy preserves the logarithmic regret order as in the centralized setting. The results apply to adaptive learning in various dynamic systems and communication networks, as well as financial investment.

💡 Research Summary

The paper tackles a restless multi‑armed bandit (RMAB) problem in which a decision‑maker must select M out of N arms at each time step, but unlike classic bandits the state of each arm evolves even when it is not played. When an arm is activated, its reward state follows an unknown Markov transition; when it is passive, it follows an arbitrary unknown stochastic process. The performance metric is regret, defined as the cumulative loss relative to a clairvoyant policy that always plays the M arms with the highest long‑run average reward.

Policy design. The authors introduce an interleaving epoch structure. Each epoch consists of a exploration phase and an exploitation phase. In the exploration phase every arm is sampled a prescribed number of times; the length of this phase grows logarithmically with the epoch index, guaranteeing that the number of observations for each arm is O(log k) after the k‑th epoch. The collected data are used to estimate both the unknown Markov transition matrix and the average reward of each arm. In the exploitation phase the policy selects the M arms with the highest estimated average reward and plays only those for the remainder of the epoch. By alternating these two phases the algorithm balances the need to learn the hidden dynamics with the desire to harvest reward.

Regret analysis with known bounds. Assuming the analyst knows non‑trivial bounds on a few system parameters (e.g., the reward range, a positive lower bound on all non‑zero transition probabilities), the authors prove that the total regret after T steps is O(log T). The proof hinges on two facts: (1) the logarithmic growth of the exploration length forces the estimation error of the transition matrix and mean reward to shrink at a rate O(1/√log k); (2) once the error is sufficiently small, the probability that the exploitation phase picks a sub‑optimal arm decays as O(1/k). Summing the exploration loss (∑ O(log k)) and the exploitation error loss (∑ O(1/k)) yields a logarithmic bound.

Regret without prior knowledge. When no a‑priori information about the system is available, the authors embed a parameter‑estimation sub‑routine inside each epoch. Early epochs use a conservative exploration length; as the algorithm gathers enough data to construct confidence intervals for the unknown bounds, it dynamically adjusts the exploration schedule to the logarithmic regime. This adaptive scheme achieves regret arbitrarily close to the optimal O(log T) – formally O((1+ε) log T) for any ε > 0 chosen by the designer.

Decentralized extension. The paper then considers a multi‑player setting where several distributed agents share the same set of arms but cannot exchange information. Two restless models are examined:

Exogenous restless: the passive dynamics are driven solely by an external environment and are independent of the players’ actions. The only source of performance loss is collisions (multiple players selecting the same arm simultaneously). The authors propose a collision‑avoidance schedule combined with random priority assignment, ensuring that each player’s exploration order is different. Under this scheme the probability of a collision decays as O(1/T), and the collective regret remains O(log T).

Endogenous restless: the passive dynamics depend on the players’ selections (e.g., a channel degrades when many users occupy it). Each player updates its own estimate of the transition matrix based on its personal history while still employing the same collision‑avoidance mechanism. The analysis shows that, despite the coupling between actions and state evolution, the decentralized algorithm retains the logarithmic regret order.

Empirical evaluation. Simulations are performed on three representative applications: (i) dynamic channel allocation in wireless networks, (ii) job scheduling in cloud computing clusters, and (iii) portfolio selection in financial markets. The proposed algorithm is benchmarked against classic UCB, Thompson Sampling, and existing RMAB heuristics. Across all scenarios the cumulative regret grows roughly as log T and is 30 %–50 % lower than the baselines, even when the algorithm starts with no knowledge of system parameters.

Conclusions and future work. The paper contributes a novel RMAB framework that simultaneously learns unknown Markovian dynamics for active arms and arbitrary stochastic dynamics for passive arms. By structuring learning into logarithmically expanding exploration epochs, it attains the optimal logarithmic regret both in centralized and fully decentralized settings. Future directions suggested include handling abrupt non‑stationary changes, incorporating limited communication among agents for semi‑centralized coordination, and extending the analysis to continuous‑state, continuous‑reward spaces.