Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descent

With the fast development of big data, learning the optimal decision rule by recursively updating it and making online decisions has been easier than before. We study the online statistical inference of model parameters in a contextual bandit framework of sequential decision-making. We propose a general framework for an online and adaptive data collection environment that can update decision rules via weighted stochastic gradient descent. We allow different weighting schemes of the stochastic gradient and establish the asymptotic normality of the parameter estimator. Our proposed estimator significantly improves the asymptotic efficiency over the previous averaged SGD approach via inverse probability weights. We also conduct an optimality analysis on the weights in a linear regression setting. We provide a Bahadur representation of the proposed estimator and show that the remainder term in the Bahadur representation entails a slower convergence rate compared to classical SGD due to the adaptive data collection.

💡 Research Summary

The paper tackles the problem of performing statistical inference for model parameters in a contextual bandit setting where data are collected adaptively based on past decisions. Traditional approaches to online learning in bandits focus on minimizing cumulative regret, while inference—providing confidence intervals or hypothesis tests for the underlying parameters—has received far less attention. A common technique for online estimation is stochastic gradient descent (SGD). When applied to bandits, the gradient computed at each round is based only on the selected arm, which introduces bias. The inverse‑probability‑weighting (IPW) scheme corrects this bias by weighting the gradient with the reciprocal of the arm‑selection probability. However, IPW inflates the variance dramatically when the exploration probability ε is small, leading to overly wide confidence intervals.

The authors propose a generalized weighted SGD framework that allows the weight (w_t) to be any positive, possibly data‑dependent function, not limited to the IPW form. The update for each arm‑specific parameter vector (\theta_{a,t}) is \