General Formulation and PCL-Analysis for Restless Bandits with Limited Observability

In this paper, we consider a general observation model for restless multi-armed bandit problems. The operation of the player is based on the past observation history that is limited (partial) and error-prone due to resource constraints or environmental or intrinsic noises. By establishing a general probabilistic model for dynamics of the observation process, we formulate the problem as a restless bandit with an infinite high-dimensional belief state space. We apply the achievable region method with partial conservation law (PCL) to the infinite-state problem and analyze its indexability and priority index (Whittle index). Finally, we propose an approximation process to transform the problem into which the AG algorithm of Niño-Mora (2001) for finite-state problems can be applied. Numerical experiments show that our algorithm has excellent performance.

💡 Research Summary

**
The paper tackles a restless multi‑armed bandit (RMAB) problem in which the decision maker receives only limited, noisy observations of the activated arms. Unlike most existing works that assume either perfect observation or a very specific error structure, the authors introduce a general observation model characterized by an arbitrary error matrix ε (the probability of observing state j when the true state is i) and a reward matrix r (the reward obtained when the true state is i and the observed state is j). They also allow for an additional feedback signal F that may depend on both the true and observed states, thereby covering a broad class of practical scenarios such as opportunistic spectrum access with acknowledgments.

The state of each arm is not directly observable; instead, the decision maker maintains a belief vector ω∈Ωₐ, where ωᵢ denotes the posterior probability that the arm is in true state i given all past observations. The belief update is derived rigorously for three cases: (i) when the arm is activated and both observation and feedback are available, (ii) when only the observation is available, and (iii) when no observation is obtained (passive case). In the active cases the update combines the transition matrix P, the error matrix ε, and the reward‑derived feedback probabilities ρ, while in the passive case the belief simply propagates through P. This results in a countable but high‑dimensional belief space, because the belief vector evolves only at discrete decision epochs.

The central theoretical contribution is the extension of the Partial Conservation Law (PCL) framework to this infinite‑state setting. Classical PCL, introduced by Niño‑Mora (2001), provides sufficient conditions under which a restless bandit is Whittle‑indexable: the existence of a Lagrangian subsidy λ for passivity such that the optimal single‑arm policy switches monotonically with λ. However, PCL was originally proved only for finite state spaces. The authors overcome this limitation by establishing a weak duality between the original RMAB and its Lagrangian relaxation, formulating both as linear programs (LPs) whose feasible region is an extended polymatroid. They then prove that, even with a countable belief space, the PCL conditions can be satisfied on appropriately constructed chains (nested subsets of the belief space). By carefully handling limiting arguments and pointwise convergence of belief updates, they demonstrate that the indexability of each arm holds, guaranteeing the existence of a Whittle index.

To make the theory computationally tractable, the infinite belief space is approximated by a finite grid Ω̃. On this discretized space the authors apply the AG algorithm of Niño‑Mora (2001), which computes the Whittle index by tracking the values of the subsidy λ at which the optimal action for a given belief switches from passive to active. The algorithm proceeds by solving a series of linear programs for each grid point, identifying the critical λ‑values (thresholds), and then interpolating between grid points to obtain a continuous index function. The paper provides error bounds for the discretization and shows that the approximation converges to the true index as the grid is refined.

Numerical experiments focus on two representative applications. The first is opportunistic spectrum access (OSA), where a user selects a channel to sense, possibly transmits data, and receives an acknowledgment (ACK) as feedback. The second involves multi‑channel communication with stochastic channel states and imperfect sensing. In both settings the proposed method is compared against baseline policies such as the classic Gittins index (which assumes perfect observation), Bayesian Thompson sampling, and simple threshold policies. Results indicate that the PCL‑based Whittle index policy consistently yields higher discounted cumulative reward, especially when observation error rates are high (e.g., >30%). Moreover, the algorithm converges rapidly, requiring far fewer simulations than sampling‑based methods.

In summary, the paper makes three major contributions: (1) a general probabilistic formulation of RMABs with arbitrary observation errors and feedback, leading to a countable belief‑state MDP; (2) a rigorous extension of the PCL framework to infinite‑dimensional belief spaces, establishing Whittle‑indexability under broad conditions; and (3) a practical computational scheme that discretizes the belief space and employs the AG algorithm to obtain accurate Whittle indices. These advances broaden the applicability of index policies to realistic systems where observations are noisy, delayed, or partially missing, and they open new avenues for efficient control of large‑scale stochastic networks.