Fast global convergence of gradient methods for high-dimensional statistical recovery

Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framework that allows the data dimension $\pdim$ to grow with (and possibly exceed) the sample size $\numobs$. This high-dimensional structure precludes the usual global assumptions—namely, strong convexity and smoothness conditions—that underlie much of classical optimization analysis. We define appropriately restricted versions of these conditions, and show that they are satisfied with high probability for various statistical models. Under these conditions, our theory guarantees that projected gradient descent has a globally geometric rate of convergence up to the \emph{statistical precision} of the model, meaning the typical distance between the true unknown parameter $\theta^*$ and an optimal solution $\hat{\theta}$. This result is substantially sharper than previous convergence results, which yielded sublinear convergence, or linear convergence only up to the noise level. Our analysis applies to a wide range of $M$-estimators and statistical models, including sparse linear regression using Lasso ($\ell_1$-regularized regression); group Lasso for block sparsity; log-linear models with regularization; low-rank matrix recovery using nuclear norm regularization; and matrix decomposition. Overall, our analysis reveals interesting connections between statistical precision and computational efficiency in high-dimensional estimation.

💡 Research Summary

The paper addresses a fundamental gap in the theory of first‑order optimization methods for high‑dimensional statistical estimation. Classical convergence analyses rely on global strong convexity and smoothness of the objective, conditions that fail when the ambient dimension d exceeds the sample size n—a regime that is now common in modern data science. In such settings, existing results for projected gradient descent (PGD) or Nesterov‑type composite gradient methods (CG) can only guarantee sub‑linear rates (typically O(1/t)) or linear convergence up to a noise‑level tolerance that is much larger than the statistical precision of the estimator.

To overcome this limitation, the authors introduce two restricted structural conditions: Restricted Strong Convexity (RSC) and Restricted Smoothness (RSM). RSC requires that the loss function exhibit sufficient curvature only on a low‑dimensional subspace that captures the true parameter’s structure (e.g., sparsity, low‑rankness). Formally, for all vectors Δ lying in a cone defined by the regularizer, the quadratic form Δᵀ∇²Lₙ(θ)Δ is bounded below by α‖Δ‖₂² minus a small tolerance term proportional to ‖Δ‖₁². RSM, on the other hand, ensures that the gradient is Lipschitz‑continuous on the same restricted set, allowing a bound of the form ‖∇Lₙ(θ₁)−∇Lₙ(θ₂)‖₂ ≤ β‖θ₁−θ₂‖₂ + γ‖θ₁−θ₂‖₁. The paper proves that for a broad family of statistical models—including sparse linear regression with random isotropic designs, group‑sparse regression, matrix completion, and robust low‑rank decomposition—both RSC and RSM hold with high probability.

Armed with these conditions, the authors analyze two simple first‑order schemes. The first is projected gradient descent applied to the constrained formulation (minimize loss subject to a norm ball). The update is θ^{t+1}=Π_{B_R(ρ)}(θ^t−η∇Lₙ(θ^t)), where Π denotes Euclidean projection onto the regularizer ball. The second is a composite gradient method (a variant of Nesterov’s proximal gradient) for the regularized formulation (minimize loss plus λR(θ)). Both algorithms use a fixed step size η and, crucially, no acceleration beyond the basic proximal step.

The main theorem shows that under RSC/RSM, the iterates converge globally and linearly to a point within statistical precision of the true parameter. Specifically, for any initialization, the error satisfies
‖θ^t−θ̂‖₂ ≤ C·(1−κ)^t·‖θ^0−θ̂‖₂ + O(ε_stat),
where θ̂ is the optimizer of the regularized problem, κ∈(0,1) is a contraction factor that depends explicitly on n, d, and the structural parameters (e.g., sparsity s), and ε_stat denotes the minimax mean‑squared error achievable by the estimator. Thus the algorithm reaches the statistical limit in a number of iterations proportional to log(1/ε_stat), which is dramatically faster than the O(1/t) rates of earlier analyses.

A particularly insightful contribution is the quantitative relationship between κ and problem dimensions. For the Lasso, κ≈c·(n/(s·log d)), indicating that as long as the sample size scales proportionally to s·log d, the convergence factor remains bounded away from zero, yielding a dimension‑independent linear rate. The authors validate this prediction empirically: they run PGD on ℓ₁‑constrained least‑squares problems with d∈{5 000,10 000,20 000} and a fixed n=2 500, observing straight lines on a log‑error versus iteration plot (geometric decay). When they rescale n according to n≈α·s·log d, the three curves collapse onto a single trajectory, confirming the theory.

The paper also clarifies the distinction between RSC used for statistical consistency (as in prior work) and the stronger RSC/RSM pair needed for optimization error control. While RSC alone suffices to bound the statistical error of the estimator, guaranteeing fast algorithmic convergence requires the additional smoothness condition on the restricted set. The authors discuss how these conditions are verified for each model class, often leveraging concentration inequalities for sub‑Gaussian designs or matrix Bernstein bounds for low‑rank problems.

In summary, the work establishes that for a wide class of high‑dimensional M‑estimators, simple first‑order methods—projected gradient descent and composite gradient descent—achieve global geometric convergence up to the intrinsic statistical accuracy of the problem. This bridges a gap between statistical theory (which provides minimax rates) and computational practice (which seeks fast algorithms), showing that the same structural assumptions that make an estimator statistically optimal also render the associated optimization tractable. The results have immediate practical implications: practitioners can rely on inexpensive gradient‑based solvers with fixed step sizes, confident that they will converge rapidly to the best possible statistical solution, even when d≫n.