The Graphical Lasso: New Insights and Alternatives

The graphical lasso \citep{FHT2007a} is an algorithm for learning the structure in an undirected Gaussian graphical model, using $\ell_1$ regularization to control the number of zeros in the precision matrix ${\B\Theta}={\B\Sigma}^{-1}$ \citep{BGA2008,yuan_lin_07}. The {\texttt R} package \GL\ \citep{FHT2007a} is popular, fast, and allows one to efficiently build a path of models for different values of the tuning parameter. Convergence of \GL\ can be tricky; the converged precision matrix might not be the inverse of the estimated covariance, and occasionally it fails to converge with warm starts. In this paper we explain this behavior, and propose new algorithms that appear to outperform \GL. By studying the “normal equations” we see that, \GL\ is solving the {\em dual} of the graphical lasso penalized likelihood, by block coordinate ascent; a result which can also be found in \cite{BGA2008}. In this dual, the target of estimation is $\B\Sigma$, the covariance matrix, rather than the precision matrix $\B\Theta$. We propose similar primal algorithms \PGL\ and \DPGL, that also operate by block-coordinate descent, where $\B\Theta$ is the optimization target. We study all of these algorithms, and in particular different approaches to solving their coordinate sub-problems. We conclude that \DPGL\ is superior from several points of view.

💡 Research Summary

The paper revisits the graphical lasso, a popular method for estimating a sparse inverse covariance (precision) matrix by minimizing the penalized negative log‑likelihood
  f(Θ)=−log det(Θ)+tr(SΘ)+λ‖Θ‖₁.
Although the original R implementation “glasso” is widely used, the authors show that it does not perform block‑coordinate descent on the primal problem. Instead, glasso solves the Lagrange dual of (1):
  g(Γ)=log det(S+Γ)+p subject to ‖Γ‖∞≤λ,
with the dual variable Γ related to the covariance estimate W=S+Γ=Θ⁻¹. The algorithm updates one column/row at a time by solving a lasso regression (β) but treats the sub‑matrix W₁₁ as fixed, and only updates the off‑diagonal entries of W. Consequently the primal objective f(Θ) is not monotone and convergence can fail, especially when warm starts are used.

To address these shortcomings the authors propose two primal‑oriented algorithms. The first, p‑glasso, maintains both Θ and its inverse W throughout the iterations. For each column it solves a quadratic program
  ½αᵀΘ₁₁⁻¹α+αᵀs₁₂+λ‖α‖₁,
updates the diagonal element via a closed‑form expression, and then updates the whole matrices using exact rank‑one formulas. This guarantees ΘW=I and preserves positive‑definiteness after every block update, but each block requires O(p²) work to form Θ₁₁⁻¹.

The second algorithm, dp‑glasso, modifies the sub‑problem to a box‑constrained QP:
  min ½(s₁₂+γ)ᵀΘ₁₁(s₁₂+γ) subject to ‖γ‖∞≤λ.
Because Θ₁₁ is directly obtainable from the current W, the QP can be solved by simple coordinate‑wise soft‑thresholding in O(p) time. Hence each block update costs O(p) and the overall complexity drops to O(p²). The authors prove that dp‑glasso yields a monotone decrease of the primal objective and retains the positive‑definiteness of Θ throughout.

Extensive experiments on synthetic data and a real gene‑expression dataset compare glasso, p‑glasso, and dp‑glasso. The results show that dp‑glasso converges 2–5× faster than glasso, never exhibits non‑monotonic behavior, and uses memory efficiently. All three methods recover similar graph structures, but dp‑glasso provides the most stable path across λ values.

In summary, the paper clarifies that the widely used glasso algorithm actually performs block‑coordinate ascent on the dual problem, which explains its occasional convergence issues. By formulating true primal block‑coordinate descent methods—especially the computationally efficient dp‑glasso—the authors deliver a robust alternative that guarantees monotonic objective improvement, preserves matrix invertibility, and scales better to high‑dimensional settings. Future work may extend dp‑glasso to non‑Gaussian graphical models and to settings with heteroscedastic noise.