Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization

Relative to the large literature on upper bounds on complexity of convex optimization, lesser attention has been paid to the fundamental hardness of these problems. Given the extensive use of convex optimization in machine learning and statistics, gaining an understanding of these complexity-theoretic issues is important. In this paper, we study the complexity of stochastic convex optimization in an oracle model of computation. We improve upon known results and obtain tight minimax complexity estimates for various function classes.

💡 Research Summary

The paper investigates the fundamental difficulty of stochastic convex optimization (SCO) by establishing information‑theoretic lower bounds on the oracle complexity required to achieve a prescribed optimization error. The authors work within a standard stochastic oracle model: at each iteration an algorithm queries an oracle that returns either a noisy function value or a stochastic subgradient sampled from an underlying distribution. While much of the literature focuses on upper bounds—showing that specific algorithms such as stochastic gradient descent (SGD), accelerated methods, or variance‑reduced schemes achieve certain rates—the lower‑bound side has received comparatively little attention, especially for a variety of function classes that appear in machine learning and statistics.

The contribution is threefold. First, the paper defines a clean, unified framework for deriving minimax lower bounds using classical tools from information theory, notably Fano’s inequality and a refined version of Le Cam’s method. The authors construct a hard family of convex functions parameterized by a finite set of binary vectors. By carefully controlling the Kullback‑Leibler divergence between the distributions induced by different functions under the stochastic oracle, they translate the statistical indistinguishability of these functions into a lower bound on the number of oracle calls required to separate them with high probability. This approach yields tight, dimension‑free results that match known upper bounds up to constant factors.

Second, the paper presents explicit lower bounds for three important classes of convex functions:

1. Strongly convex, L‑Lipschitz functions – For functions that are μ‑strongly convex and have L‑Lipschitz gradients, any algorithm that guarantees an expected suboptimality of at most ε must make at least
   \