Data-driven Analysis of First-Order Methods via Distributionally Robust Optimization

We consider the problem of analyzing the probabilistic performance of first-order methods when solving convex optimization problems drawn from an unknown distribution only accessible through samples. By combining performance estimation (PEP) and Wasserstein distributionally robust optimization (DRO), we formulate the analysis as a tractable semidefinite program. Our approach unifies worst-case and average-case analyses by incorporating data-driven information from the observed convergence of first-order methods on a limited number of problem instances. This yields probabilistic, data-driven performance guarantees in terms of the expectation or conditional value-at-risk of the selected performance metric. Experiments on smooth convex minimization, logistics regression, and Lasso show that our method significantly reduces the conservatism of classical worst-case bounds and narrows the gap between theoretical and empirical performance.

💡 Research Summary

The paper tackles the long‑standing gap between worst‑case convergence guarantees for first‑order optimization algorithms and the often much better empirical performance observed on real‑world problem instances. To bridge this gap, the authors propose a novel framework that unifies Performance Estimation Problems (PEP) with Wasserstein‑based Distributionally Robust Optimization (DRO).

PEP is a powerful tool that encodes the K‑step behavior of a deterministic first‑order method as a semidefinite program (SDP). By fixing a function class F (e.g., L‑smooth, µ‑strongly convex) and an initial‑condition set X(f), the worst‑case value of a scalar performance metric ϕ_K (such as function‑value gap, distance to solution, or gradient norm) can be expressed as the optimal value of an SDP. This approach has already yielded tight worst‑case rates and even led to the discovery of optimal accelerated schemes (e.g., OGM).

However, classical PEP only yields a bound that holds for all functions in F, regardless of how likely those functions are to appear in practice. The authors therefore introduce a data‑driven component: they assume that problem instances (f, x⁰) are drawn from an unknown distribution P supported on H = F × ℝ^d, but only a finite sample {(f_i, x⁰_i)}_{i=1}^N is available. Using the empirical distribution (\hat P_N) as a center, they construct a Wasserstein ambiguity set
\