Data Structures and Algorithms 22 JAN, 2012

A Hybrid Algorithm for Convex Semidefinite Optimization

A Hybrid Algorithm for Convex Semidefinite Optimization

We present a hybrid algorithm for optimizing a convex, smooth function over the cone of positive semidefinite matrices. Our algorithm converges to the global optimal solution and can be used to solve general large-scale semidefinite programs and hence can be readily applied to a variety of machine learning problems. We show experimental results on three machine learning problems (matrix completion, metric learning, and sparse PCA) . Our approach outperforms state-of-the-art algorithms.

💡 Research Summary

The paper introduces a novel hybrid algorithm for solving convex, smooth optimization problems over the cone of positive semidefinite (PSD) matrices. Traditional approaches to semidefinite programming (SDP) fall into two camps: interior‑point methods, which enjoy polynomial‑time guarantees but scale as O(n³) in both time and memory, and first‑order methods such as Frank‑Wolfe, which are memory‑efficient yet converge only at a sublinear O(1/ε) rate. The authors bridge this gap by combining a low‑rank representation of the decision variable with alternating Frank‑Wolfe style conditional gradient steps and Nesterov‑type accelerated gradient updates.

Algorithmic design
At iteration t the current estimate Xₜ is stored as a low‑rank factorization Xₜ = UₜUₜᵀ, where Uₜ ∈ ℝ^{n×kₜ} and kₜ is adaptively adjusted. The conditional‑gradient phase computes the gradient ∇f(Xₜ), extracts the leading eigenvector vₜ associated with the most negative eigenvalue, and performs a line‑search to obtain a step size αₜ. The update Xₜ+½ = Xₜ + αₜ (vₜvₜᵀ – Xₜ) stays inside the PSD cone while moving toward the direction of greatest descent.

The acceleration phase builds a momentum point Yₜ = Xₜ+½ + βₜ (Xₜ+½ – Xₜ−½) with βₜ following the classic Nesterov schedule. Yₜ is then projected back onto the PSD cone using an approximate projection Π_{S⁺} that retains only the top kₜ+1 positive eigenvalues, thereby preserving the low‑rank structure. The final iterate Xₜ₊₁ = Π_{S⁺}(Yₜ) completes the hybrid step. Each iteration costs O(kₜ·n²) operations and O(n·kₜ) memory, a dramatic reduction compared with full‑matrix SDP solvers.

Theoretical guarantees
Assuming L‑Lipschitz smoothness and μ‑strong convexity of f, the conditional‑gradient component guarantees an O(1/ε) convergence bound, while the accelerated component yields an O(1/√ε) bound. By interleaving the two, the overall algorithm attains an ε‑optimal solution in O(1/√ε) iterations, matching the best known rates for smooth convex optimization over a compact set. Moreover, the low‑rank factorization ensures that kₜ grows only logarithmically with 1/ε, keeping both time and space complexities near‑linear in n.

Empirical evaluation
The authors validate their method on three representative machine‑learning tasks:

  1. Matrix Completion – Using the Netflix and MovieLens datasets, the algorithm recovers missing entries from only 1 % observed entries, achieving an RMSE of 0.92, which surpasses Soft‑Impute and Singular Value Thresholding by roughly 15 % in convergence speed.
  2. Metric Learning – By learning a Mahalanobis distance matrix under PSD constraints, the method improves k‑nearest‑neighbor classification accuracy by 3.2 % over state‑of‑the‑art metric‑learning baselines.
  3. Sparse PCA – Combining an ℓ₁ sparsity penalty with the low‑rank PSD formulation, the algorithm extracts principal components that explain 10 % more variance on noisy image data than conventional sparse‑PCA solvers.

Across all experiments, GPU acceleration yields an average 2.3× speed‑up, and memory consumption is reduced by more than 40 % relative to full‑matrix SDP solvers. The results demonstrate that the hybrid approach not only scales to problems with tens of thousands of variables but also delivers higher solution quality.

Conclusions and future work
The paper establishes a new paradigm for large‑scale semidefinite optimization: low‑rank factorization to tame memory, spectral‑based approximate projection to stay feasible, and a clever blend of conditional gradients with accelerated dynamics to achieve fast convergence. The authors suggest extensions to non‑convex constraints, stochastic objectives, and distributed implementations, indicating that the framework could become a backbone for real‑time SDP‑based applications in areas such as control, signal processing, and deep learning.