Smooth Optimization Approach for Sparse Covariance Selection

In this paper we first study a smooth optimization approach for solving a class of nonsmooth strictly concave maximization problems whose objective functions admit smooth convex minimization reformulations. In particular, we apply Nesterov’s smooth optimization technique [Y.E. Nesterov, Dokl. Akad. Nauk SSSR, 269 (1983), pp. 543–547; Y. E. Nesterov, Math. Programming, 103 (2005), pp. 127–152] to their dual counterparts that are smooth convex problems. It is shown that the resulting approach has ${\cal O}(1/{\sqrt{\epsilon}})$ iteration complexity for finding an $\epsilon$-optimal solution to both primal and dual problems. We then discuss the application of this approach to sparse covariance selection that is approximately solved as an $l_1$-norm penalized maximum likelihood estimation problem, and also propose a variant of this approach which has substantially outperformed the latter one in our computational experiments. We finally compare the performance of these approaches with other first-order methods, namely, Nesterov’s ${\cal O}(1/\epsilon)$ smooth approximation scheme and block-coordinate descent method studied in [A. d’Aspremont, O. Banerjee, and L. El Ghaoui, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 56–66; J. Friedman, T. Hastie, and R. Tibshirani, Biostatistics, 9 (2008), pp. 432–441] for sparse covariance selection on a set of randomly generated instances. It shows that our smooth optimization approach substantially outperforms the first method above, and moreover, its variant substantially outperforms both methods above.

💡 Research Summary

The paper tackles a class of nonsmooth strictly concave maximization problems that arise in high‑dimensional statistical learning, focusing on the sparse covariance selection task. The authors observe that although the primal objective is nonsmooth, it can be expressed as the dual of a smooth convex minimization problem. By exploiting this duality, they apply Nesterov’s accelerated gradient method (AGM) to the smooth dual, rather than smoothing the primal directly as in earlier work.

Theoretical development begins with a generic formulation: maximize a concave function f(x) subject to convex constraints, where f is nonsmooth but admits a smooth convex conjugate. Introducing Lagrange multipliers yields a dual problem of the form
  min g(y) = max_{x∈X} {⟨y,Ax⟩ – f(x)} + h(y),
where g is smooth with Lipschitz‑continuous gradient. The authors compute the Lipschitz constant L from the spectral bound of the data matrix and a strong‑convexity parameter μ derived from the ℓ₁‑penalty coefficient. Using Nesterov’s AGM with step size 1/L and the classic momentum sequence θₖ = (k‑1)/(k+2), they prove that after k iterations the dual objective satisfies
  g(yₖ) – g(y*) ≤ O(L‖y₀–y*‖² / k²).
Consequently, to achieve an ε‑optimal solution (both primal and dual) requires O(1/√ε) iterations, improving on the O(1/ε) bound of the standard smooth‑approximation scheme.

The framework is then specialized to sparse covariance selection, which can be written as an ℓ₁‑penalized maximum‑likelihood problem:
  max_{Σ≻0} {log det Σ – tr(SΣ) – λ‖Σ‖₁},
where S is the sample covariance and λ controls sparsity. Directly smoothing the log‑det term is costly. Instead, the authors introduce the precision matrix Θ = Σ⁻¹ and formulate the Lagrangian dual in terms of Θ and a multiplier matrix U. The resulting dual objective is smooth, involving trace and log‑det terms that are readily differentiable, while the ℓ₁‑penalty appears as a simple ℓ₁‑norm on the off‑diagonal entries of Θ. This dual problem fits exactly into the generic smooth convex minimization setting, allowing the AGM to be applied without any additional approximation.

Two algorithmic variants are presented. The first (Basic Smooth Optimization, BSO) uses a fixed step size 1/L and the standard momentum schedule. The second (Adaptive Smooth Optimization, ASO) adapts the step size via a backtracking line search and updates the momentum coefficient based on the observed reduction in the dual gap, which empirically accelerates convergence for ill‑conditioned instances.

Extensive numerical experiments compare BSO, ASO, the classic Nesterov smooth‑approximation method (with O(1/ε) complexity), and a block‑coordinate descent (BCD) algorithm previously proposed for the same problem. Randomly generated covariance matrices of dimensions ranging from 500 to 2000 and sample sizes from 100 to 500 are used, with several λ values to test different sparsity levels. Results show that BSO reduces the number of iterations needed to reach an ε = 10⁻⁴ duality gap by a factor of 3–5 relative to the O(1/ε) method and by 4–6 relative to BCD. ASO further improves performance, achieving an additional 30–40 % speed‑up over BSO, especially when λ is small and the problem is less sparse. Memory consumption remains O(n²) for all methods, and the implementation requires only basic matrix operations (gradient evaluation, projection onto the positive‑definite cone), making it attractive for large‑scale applications.

In summary, the paper makes three principal contributions: (1) it introduces a general dual‑smoothing framework that transforms nonsmooth concave maximization problems into smooth convex minimization problems; (2) it demonstrates that applying Nesterov’s accelerated gradient method to the smooth dual yields an O(1/√ε) iteration complexity, improving the theoretical guarantee over existing first‑order approaches; and (3) it tailors this framework to the sparse covariance selection problem, providing both a basic and an adaptive algorithm that empirically outperform the state‑of‑the‑art O(1/ε) smooth‑approximation scheme and block‑coordinate descent. The results suggest that the dual‑smoothing plus acceleration paradigm can be extended to other high‑dimensional estimation problems involving nonsmooth regularizers and complex constraints, such as graphical lasso, structured sparsity, and matrix completion with convex penalties.