Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design

Many applications require optimizing an unknown, noisy function that is expensive to evaluate. We formalize this task as a multi-armed bandit problem, where the payoff function is either sampled from a Gaussian process (GP) or has low RKHS norm. We resolve the important open problem of deriving regret bounds for this setting, which imply novel convergence rates for GP optimization. We analyze GP-UCB, an intuitive upper-confidence based algorithm, and bound its cumulative regret in terms of maximal information gain, establishing a novel connection between GP optimization and experimental design. Moreover, by bounding the latter in terms of operator spectra, we obtain explicit sublinear regret bounds for many commonly used covariance functions. In some important cases, our bounds have surprisingly weak dependence on the dimensionality. In our experiments on real sensor data, GP-UCB compares favorably with other heuristical GP optimization approaches.

💡 Research Summary

The paper addresses the problem of sequentially optimizing an unknown, noisy, and expensive-to-evaluate function by casting it as a stochastic multi‑armed bandit problem. The authors consider two complementary assumptions on the target function f: (i) f is a sample from a Gaussian Process (GP) with known mean zero and covariance kernel k, or (ii) f belongs to the reproducing kernel Hilbert space (RKHS) associated with k and has bounded RKHS norm. Both assumptions encode smoothness in a non‑parametric way, allowing the framework to handle high‑dimensional, non‑linear functions that arise in applications such as sensor placement, ad selection, or robot control.

The central contribution is the analysis of the GP‑UCB (Gaussian Process Upper Confidence Bound) algorithm, a Bayesian analogue of the classic UCB strategy for finite‑armed bandits. At each round t, GP‑UCB computes the posterior mean μ_{t‑1}(x) and standard deviation σ_{t‑1}(x) for every candidate point x∈D, then selects
 x_t = argmax_{x∈D} μ_{t‑1}(x) + √β_t σ_{t‑1}(x).
The exploration parameter β_t grows logarithmically with t, guaranteeing that with high probability the true function value is bounded above by the UCB index. This rule balances exploration (large posterior variance) and exploitation (large posterior mean) in a principled way.

The authors derive a regret bound of the form
 R_T = Õ(√{T β_T γ_T}),
where γ_T is the maximum information gain after T observations:
 γ_T = max_{A⊂D, |A|=T} I(y_A; f) = ½ log|I + σ^{-2}K_A|.
Here I(y_A; f) denotes the mutual information between the noisy observations y_A and the underlying function f, and K_A is the kernel matrix on the selected points. This formulation establishes a novel connection between bandit regret analysis and Bayesian experimental design, because the same information‑gain functional governs optimal sensor placement and active learning.

A key technical step is bounding γ_T for common kernels using spectral properties of the associated integral operator. For the linear kernel, γ_T = O(d log T), leading to a regret of Õ(√{Td log T}). For the squared‑exponential (RBF) kernel, γ_T = O((log T)^{d+1}), which yields a regret that depends only polylogarithmically on the dimension d—a striking improvement over linear‑bandit bounds that scale as √{Td}. For Matérn kernels with smoothness parameter ν, the bound becomes γ_T = O(T^{d(d+1)/(2ν + d(d+1))} log T); larger ν (smoother functions) reduces the dimensional dependence. These results demonstrate that by choosing an appropriate kernel, one can obtain sublinear regret even in high‑dimensional spaces.

The analysis also extends to the agnostic setting where f is not drawn from a GP but merely satisfies ‖f‖_k ≤ B. Using concentration inequalities for sub‑Gaussian noise and properties of the RKHS, the same Õ(√{T β_T γ_T}) regret bound holds, providing distribution‑free guarantees.

Empirically, the authors evaluate GP‑UCB on a real‑world temperature‑sensor network. They compare against popular Bayesian optimization heuristics such as Expected Improvement (EI), Probability of Improvement (PI), and a pure information‑gain greedy strategy. GP‑UCB consistently finds higher temperature values with fewer samples and exhibits lower cumulative regret, confirming the practical relevance of the theoretical findings. The experiments also illustrate that the algorithm remains effective when the input space is continuous, using global optimization heuristics to approximate the argmax in the UCB acquisition function.

In summary, the paper makes three major contributions: (1) it introduces and rigorously analyzes GP‑UCB, a simple yet powerful algorithm for GP‑based bandit optimization; (2) it links cumulative regret to the maximal information gain, thereby bridging stochastic bandits and Bayesian experimental design; and (3) it provides explicit, kernel‑dependent regret bounds that often have only logarithmic dependence on dimensionality, a substantial improvement over prior linear‑bandit results. These insights open the door to theoretically grounded, scalable optimization of expensive black‑box functions in many scientific and engineering domains.