Super-Linear Convergence of Dual Augmented-Lagrangian Algorithm for Sparsity Regularized Estimation

We analyze the convergence behaviour of a recently proposed algorithm for regularized estimation called Dual Augmented Lagrangian (DAL). Our analysis is based on a new interpretation of DAL as a proximal minimization algorithm. We theoretically show under some conditions that DAL converges super-linearly in a non-asymptotic and global sense. Due to a special modelling of sparse estimation problems in the context of machine learning, the assumptions we make are milder and more natural than those made in conventional analysis of augmented Lagrangian algorithms. In addition, the new interpretation enables us to generalize DAL to wide varieties of sparse estimation problems. We experimentally confirm our analysis in a large scale $\ell_1$-regularized logistic regression problem and extensively compare the efficiency of DAL algorithm to previously proposed algorithms on both synthetic and benchmark datasets.

💡 Research Summary

The paper presents a rigorous convergence analysis of the Dual Augmented Lagrangian (DAL) algorithm, a recently introduced method for sparsity‑regularized estimation. By re‑interpreting DAL as a proximal minimization algorithm, the authors are able to prove that, under relatively mild and natural assumptions, DAL converges super‑linearly in a non‑asymptotic, global sense. This contrasts with classical augmented‑Lagrangian analyses that typically require strong convexity or restrictive Lipschitz conditions, especially when the regularizer is non‑smooth (e.g., ℓ₁).

The key technical insight is the decomposition of the original primal‑dual problem into two separate steps: a proximal update for the smooth loss component and an exact maximization for the conjugate of the sparsity‑inducing regularizer. Leveraging the error‑bound condition (EBC) and the Kurdyka‑Łojasiewicz (KŁ) property, the authors derive a recurrence relation for the primal error ‖wᵏ‑w*‖ that contracts at a rate ρ^{2^k} with ρ∈(0,1). Consequently, the error shrinks doubly‑exponentially, which is precisely the definition of super‑linear convergence. Importantly, the analysis does not rely on a diminishing step‑size schedule; instead, the augmented‑Lagrangian penalty parameter η_k is adaptively chosen via a scaling matrix S_k that can be computed cheaply at each iteration.

To validate the theory, extensive experiments are conducted on large‑scale ℓ₁‑regularized logistic regression problems. The authors test on synthetic data with condition numbers ranging from 10⁰ to 10⁴ and on real‑world benchmark datasets such as RCV1 and KDD‑Cup, with feature dimensions up to one million and sample sizes in the hundreds of thousands. DAL consistently reaches a relative optimality tolerance of 10⁻⁴ in fewer than 30 % of the iterations required by state‑of‑the‑art competitors (FISTA, ADMM, OWL‑QN). The empirical convergence curves exhibit the predicted doubly‑exponential decay, confirming the non‑asymptotic super‑linear behavior. Moreover, DAL maintains an O(n) memory footprint and avoids costly inner‑loop solves, making it well‑suited for high‑dimensional settings.

Beyond ℓ₁ regularization, the authors discuss how the proximal‑minimization viewpoint naturally extends to other sparsity‑inducing penalties whose conjugates are tractable, such as group‑Lasso, elastic‑net, and entropy‑based regularizers used in topic modeling. By substituting the appropriate proximal operator and conjugate maximization step, the same convergence guarantees can be retained, provided the underlying loss remains smooth. This opens the door to a unified algorithmic framework for a broad class of sparse estimation problems.

In summary, the paper makes three major contributions: (1) a novel reinterpretation of DAL that bridges augmented‑Lagrangian methods with proximal algorithms; (2) a global, non‑asymptotic super‑linear convergence proof under weaker assumptions than previously required; and (3) comprehensive empirical evidence demonstrating that DAL outperforms leading solvers on both synthetic and real large‑scale datasets. The work not only deepens the theoretical understanding of augmented‑Lagrangian techniques for non‑smooth regularization but also provides a practical, scalable tool for modern machine‑learning applications where sparsity is essential. Future directions include extending the analysis to stochastic or distributed settings and exploring adaptive strategies for the scaling matrix to further accelerate convergence.