Bias-Optimal Bounds for SGD: A Computer-Aided Lyapunov Analysis

The non-asymptotic analysis of Stochastic Gradient Descent (SGD) typically yields bounds that decompose into a bias term and a variance term. In this work, we focus on the bias component and study the extent to which SGD can match the optimal convergence behavior of deterministic gradient descent. Assuming only (strong) convexity and smoothness of the objective, we derive new bounds that are bias-optimal, in the sense that the bias term coincides with the worst-case rate of gradient descent. Our results hold for the full range of constant step-sizes $γL \in (0,2)$, including critical and large step-size regimes that were previously unexplored without additional variance assumptions. The bounds are obtained through the construction of a simple Lyapunov energy whose monotonicity yields sharp convergence guarantees. To design the parameters of this energy, we employ the Performance Estimation Problem framework, which we also use to provide numerical evidence for the optimality of the associated variance terms.

💡 Research Summary

This paper addresses a fundamental gap in the non‑asymptotic analysis of Stochastic Gradient Descent (SGD): while most existing bounds decompose into a bias term that vanishes as the number of iterations grows and a variance term that can be reduced by shrinking the step size, it has remained unclear whether the bias component of SGD can ever match the optimal worst‑case convergence rate of deterministic Gradient Descent (GD). The authors introduce the notion of a “bias‑optimal” bound, defined as a bound whose bias term coincides with the worst‑case rate of GD (i.e., the rate obtained when the stochastic gradient variance at the optimum, σ*_2, is zero).

To achieve bias‑optimality, the authors construct a novel Lyapunov energy function
E_t = a_t‖x_t – x*‖² + ρ·t – Σ_{s=0}^{t‑1} f(x_s) + Σ_{s=0}^{t‑1} e_s σ*_2,
where a_t, ρ, and e_s are non‑negative parameters to be chosen. The first term is the classic distance‑to‑optimum term; the second term replaces the usual t·(f(x_t) – f*) with the sum of past function gaps, a design that, according to the authors, is new in stochastic optimization and crucial for obtaining tight bias bounds. The third term is a negative cumulative sum that compensates for stochastic fluctuations, allowing the Lyapunov function to be monotone in expectation.

The key technical contribution lies in the systematic selection of the parameters (a_t, ρ, e_s). The authors employ the Performance Estimation Problem (PEP) framework, originally introduced for deterministic first‑order methods, to cast the admissibility conditions of the Lyapunov function as a semidefinite program (SDP). Solving this SDP yields parameter sequences that both guarantee monotonicity of E_t and minimize the bias coefficient. The PEP approach also provides a numerical platform to assess the tightness of the resulting variance term.

The paper presents explicit bias‑optimal bounds for the full range of constant step‑sizes γL ∈ (0,2), where L is the smoothness constant. Three regimes are distinguished:

1. Small step‑sizes (γL ∈ (0,1)):
   Bias(T) ≈ (1/(2γ))·T⁻¹,
   Variance(T) ≈ γ²/(1‑γL).
   The bias matches the GD worst‑case rate, and the constant improves upon prior results.

2. Critical step‑size (γL = 1):
   Bias(T) can be made arbitrarily close to L²·T⁻¹ (the limiting optimal bias as γL → 1), but this requires a variance term that diverges as ε → 0:
   Variance(T) ≈ γ(2+ε)/(ε(2‑ε)).
   The authors conjecture that a finite variance bound cannot coexist with the exact optimal bias at γL = 1, a claim supported by extensive numerical experiments.

3. Large step‑sizes (γL ∈ (1,2)):
   Bias(T) ≈ (1/(2γ(2‑γL)))·T⁻¹, again matching the GD worst‑case rate.
   However, the variance term grows exponentially with T:
   Variance(T) ≈ exp(T)/(2‑γL).
   This exponential growth is a novel phenomenon not observed in previous SGD analyses. The authors show that if one relaxes the bias slightly (allowing a sub‑optimal bias), the variance can be kept uniformly bounded (Lemma 4.12).

The analysis is then extended to the strongly convex case (µ > 0). The performance metric becomes the expected squared distance to the optimum, Δ(T) = E‖x_T – x*‖². For non‑critical step‑sizes, the bias term is Bias(T) = ϕ²·T⁻¹, where ϕ = max{1‑γµ, γL‑1} is the optimal GD contraction factor. The critical step‑size γ = 2µ + L exhibits the same singular behavior as in the convex case: the bias can be approached arbitrarily closely, but only at the cost of an unbounded variance term.

Beyond SGD, the authors apply their Lyapunov‑PEP methodology to the stochastic proximal algorithm, thereby obtaining the first bias‑optimal guarantees for a broad class of convex, possibly nonsmooth problems where the objective may have a restricted domain.

The paper situates its contributions within a rich literature. Early SGD analyses relied on strong variance or gradient boundedness assumptions (e.g., uniform bounded variance, bounded gradients). More recent works have progressively weakened these assumptions, often requiring a “variance transfer” inequality that bounds the stochastic gradient variance in terms of σ*_2 and problem geometry. The present work operates under the minimal assumption of L‑smoothness and (strong) convexity, which already imply a suitable variance transfer inequality, and thus fits naturally into the modern “no‑variance‑assumption” paradigm.

In summary, the authors deliver a unified, computer‑aided framework that (i) constructs a simple yet powerful Lyapunov function, (ii) leverages PEP to select optimal parameters, and (iii) yields bias‑optimal non‑asymptotic bounds for SGD across the entire admissible constant step‑size interval. The results sharpen existing bounds for small step‑sizes, uncover previously unknown interactions between bias and variance at critical and large step‑sizes, and open new avenues for algorithm design that can balance bias optimality against variance growth. The paper’s blend of rigorous analysis, semidefinite programming, and extensive numerical validation makes it a significant contribution to the theory of stochastic first‑order methods.