Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method

We give a new approach to the dictionary learning (also known as “sparse coding”) problem of recovering an unknown $n\times m$ matrix $A$ (for $m \geq n$) from examples of the form [ y = Ax + e, ] where $x$ is a random vector in $\mathbb R^m$ with at most $\tau m$ nonzero coordinates, and $e$ is a random noise vector in $\mathbb R^n$ with bounded magnitude. For the case $m=O(n)$, our algorithm recovers every column of $A$ within arbitrarily good constant accuracy in time $m^{O(\log m/\log(\tau^{-1}))}$, in particular achieving polynomial time if $\tau = m^{-\delta}$ for any $\delta>0$, and time $m^{O(\log m)}$ if $\tau$ is (a sufficiently small) constant. Prior algorithms with comparable assumptions on the distribution required the vector $x$ to be much sparser—at most $\sqrt{n}$ nonzero coordinates—and there were intrinsic barriers preventing these algorithms from applying for denser $x$. We achieve this by designing an algorithm for noisy tensor decomposition that can recover, under quite general conditions, an approximate rank-one decomposition of a tensor $T$, given access to a tensor $T’$ that is $\tau$-close to $T$ in the spectral norm (when considered as a matrix). To our knowledge, this is the first algorithm for tensor decomposition that works in the constant spectral-norm noise regime, where there is no guarantee that the local optima of $T$ and $T’$ have similar structures. Our algorithm is based on a novel approach to using and analyzing the Sum of Squares semidefinite programming hierarchy (Parrilo 2000, Lasserre 2001), and it can be viewed as an indication of the utility of this very general and powerful tool for unsupervised learning problems.

💡 Research Summary

The paper tackles the classic dictionary learning problem—recovering an unknown n × m matrix A (the “dictionary”) from observations y = Ax + e where the coefficient vector x is sparse and e is bounded noise. Prior provable algorithms required the sparsity level of x to be at most O(√n) non‑zero entries, and could not handle denser signals even with quasipolynomial time. Barak, Kelner and Steurer introduce a fundamentally different approach based on the Sum‑of‑Squares (SOS) semidefinite programming hierarchy, which enables recovery when x may have up to τ m non‑zero entries for τ as large as a small constant (or τ = m^{−δ} for any δ > 0).

The authors formalize a new class of coefficient distributions called (d, τ)-nice. Roughly, a distribution is (d, τ)-nice if (i) each coordinate’s d‑th moment is normalized to 1, (ii) all non‑square monomials of degree ≤ d have zero expectation (ensuring symmetry around zero), and (iii) for any pair of coordinates i, j the product of their d/2‑th moments is at most τ. This captures the Bernoulli‑Gaussian mixture model and many other realistic sparse‑plus‑noise scenarios, while allowing the coordinates to be correlated as long as the correlation does not exceed τ.

The core technical contribution is a noisy tensor decomposition algorithm. Consider the degree‑d homogeneous polynomial T(u) = ‖Aᵀu‖_d^d, whose rank‑1 components correspond exactly to the columns of A. The learner receives a polynomial P that is τ‑close to T in spectral norm, i.e., |P(u) − ‖Aᵀu‖_d^d| ≤ τ‖u‖_2^d for all unit vectors u. This model permits very large pointwise noise because ‖Aᵀu‖_d^d is typically of order m · n^{−d/2}, which can be far smaller than a constant τ. Consequently, P may have many spurious local minima unrelated to the true tensor, rendering local‑search methods ineffective.

The SOS algorithm circumvents this by solving a degree‑O(d) semidefinite program that captures all low‑degree moment constraints implied by the “nice” distribution. The SDP yields a pseudo‑distribution over candidate vectors u. By extracting higher‑order moments from this pseudo‑distribution (essentially performing a moment‑matching step), the algorithm isolates vectors that have unusually large d‑norm relative to their 2‑norm—precisely the directions aligned with the dictionary columns. The procedure then refines these directions to obtain ε‑accurate estimates of each column, up to permutation and scaling.

Complexity depends on τ. If τ = m^{−δ} for some constant δ > 0, the required degree d is O(log m / log τ^{−1}), and the SOS SDP can be solved in polynomial time, yielding a fully polynomial‑time algorithm for dictionaries with m = O(n). If τ is a small constant, d must be O(log m), leading to a quasipolynomial runtime of n^{O(log n)}. The algorithm works for overcomplete dictionaries (m > n) and makes no incoherence assumptions on the columns of A.

The paper also discusses how noise e can be absorbed into the coefficient distribution: since y = Ax + e can be rewritten as y = A(x + e′) for an appropriately scaled e′, the overall distribution of x + e′ remains (d, τ)-nice provided the noise magnitude is bounded. Hence the method is robust to additive noise without explicit modeling.

Compared to earlier works that rely on alternating minimization, convex relaxations, or spectral methods, the SOS‑based approach provides the first provable guarantee for dense sparse coding under constant‑level noise. It also demonstrates the broader potential of SOS hierarchies for unsupervised learning tasks, where global polynomial constraints can replace fragile local‑optimality arguments.

Limitations include the growth of the SDP size with the degree d, which may be prohibitive for very large n in practice, and the reliance on moment conditions that, while mild, may not hold for all real‑world data. Future directions suggested are (i) reducing the required degree via tighter analyses or problem‑specific relaxations, (ii) empirical validation of the (d, τ)-nice model on natural datasets, and (iii) extending the framework to other latent‑variable models where noisy tensor decompositions arise.

In summary, the authors present a novel SOS‑driven algorithm that achieves approximate dictionary recovery for substantially denser sparse codes than previously possible, tolerates constant‑size spectral‑norm noise, and opens a new line of research on applying powerful semidefinite hierarchies to fundamental unsupervised learning problems.