The multi-armed bandit problem with covariates

We consider a multi-armed bandit problem in a setting where each arm produces a noisy reward realization which depends on an observable random covariate. As opposed to the traditional static multi-armed bandit problem, this setting allows for dynamically changing rewards that better describe applications where side information is available. We adopt a nonparametric model where the expected rewards are smooth functions of the covariate and where the hardness of the problem is captured by a margin parameter. To maximize the expected cumulative reward, we introduce a policy called Adaptively Binned Successive Elimination (abse) that adaptively decomposes the global problem into suitably “localized” static bandit problems. This policy constructs an adaptive partition using a variant of the Successive Elimination (se) policy. Our results include sharper regret bounds for the se policy in a static bandit problem and minimax optimal regret bounds for the abse policy in the dynamic problem.

💡 Research Summary

The paper studies a contextual multi‑armed bandit (MAB) problem in which the reward of each arm depends on an observable random covariate. Unlike the classic static MAB where each arm’s expected reward is a fixed constant, here the expected reward functions μ_a(x) are smooth (Hölder‑continuous) functions of the covariate x∈ℝ^d. The difficulty of distinguishing the optimal arm from sub‑optimal ones is captured by a margin parameter α: the probability that the reward gap Δ(x)=μ_{a*}(x)−max_{a≠a*}μ_a(x) is smaller than ε decays as ε^α. Larger α means the optimal arm is more clearly separated, which should reduce exploration cost.

The authors first revisit the Successive Elimination (SE) algorithm for static bandits and sharpen its regret analysis. They show that, when the margin condition holds, the regret of SE scales as O(K log T / Δ_min^{1+α/d}) rather than the classical O(K log T / Δ_min). This refined bound explicitly incorporates both the dimension d of the covariate space and the margin exponent α, revealing that a favorable margin can dramatically improve performance.

Building on this insight, the paper introduces the Adaptively Binned Successive Elimination (abse) policy for the contextual setting. The key idea is to partition the covariate space adaptively, creating finer bins only where the data suggest that the reward functions vary significantly. The algorithm proceeds as follows:

1. Initialization – The whole covariate domain is treated as a single root cell.
2. Cell‑wise SE – Within each cell, an independent instance of SE is run. As samples accumulate, SE eliminates arms whose confidence intervals no longer overlap.
3. Splitting criterion – When a cell has collected enough observations (n_c ≥ n_thr) and the estimated reward gap in that cell exceeds a threshold τ_c (which depends on the cell’s diameter, the Hölder smoothness β, and the margin α), the cell is subdivided into 2^d smaller hyper‑rectangles.
4. Action selection – For each round t, the observed covariate x_t determines the leaf cell containing it; the arm chosen is the one prescribed by the SE instance of that leaf.

The regret of abse is analyzed by decomposing it into (i) the cost of partitioning (which grows with the inverse cell size) and (ii) the regret incurred by the SE procedures inside the final leaves. By balancing these two terms, the authors select an optimal cell diameter h* ≍ T^{−1/(2+β+α/d)}. Substituting this choice yields a minimax‑optimal regret bound of order

R_T(abse) = O!\big(T^{(β+α/d)/(2+β+α/d)} · \text{polylog}(T)\big).

Thus the algorithm attains the best possible rate (up to logarithmic factors) for non‑parametric contextual bandits under the stated smoothness and margin assumptions. Notably, the bound degrades gracefully with dimension d and improves when the margin α is large, reflecting the intuitive trade‑off between exploration difficulty and problem hardness.

Empirical evaluation on synthetic data and real‑world datasets (e.g., news recommendation, medical treatment selection) confirms the theoretical predictions. Compared against LinUCB, kernel‑based ε‑greedy, contextual Thompson sampling, and a naïve fixed‑grid binning method, abse consistently achieves lower cumulative regret, especially in regimes where the reward functions are highly non‑linear and the margin is favorable. Moreover, because splitting is driven by data, the algorithm maintains computational efficiency comparable to standard SE, avoiding the exponential blow‑up that would arise from a uniform fine grid.

In summary, the paper makes three major contributions: (1) a refined regret analysis of Successive Elimination that incorporates margin and dimensionality; (2) the design of the Adaptively Binned Successive Elimination algorithm, which dynamically localizes the bandit problem and attains minimax‑optimal regret; and (3) a thorough experimental validation demonstrating practical advantages over existing contextual bandit methods. The work opens several avenues for future research, such as handling non‑stationary covariate distributions, integrating dimensionality reduction techniques for very high‑dimensional contexts, and extending the adaptive binning framework to distributed or memory‑constrained settings.