More Algorithms for Provable Dictionary Learning

In dictionary learning, also known as sparse coding, the algorithm is given samples of the form $y = Ax$ where $x\in \mathbb{R}^m$ is an unknown random sparse vector and $A$ is an unknown dictionary matrix in $\mathbb{R}^{n\times m}$ (usually $m > n$, which is the overcomplete case). The goal is to learn $A$ and $x$. This problem has been studied in neuroscience, machine learning, visions, and image processing. In practice it is solved by heuristic algorithms and provable algorithms seemed hard to find. Recently, provable algorithms were found that work if the unknown feature vector $x$ is $\sqrt{n}$-sparse or even sparser. Spielman et al. \cite{DBLP:journals/jmlr/SpielmanWW12} did this for dictionaries where $m=n$; Arora et al. \cite{AGM} gave an algorithm for overcomplete ($m >n$) and incoherent matrices $A$; and Agarwal et al. \cite{DBLP:journals/corr/AgarwalAN13} handled a similar case but with weaker guarantees. This raised the problem of designing provable algorithms that allow sparsity $\gg \sqrt{n}$ in the hidden vector $x$. The current paper designs algorithms that allow sparsity up to $n/poly(\log n)$. It works for a class of matrices where features are individually recoverable, a new notion identified in this paper that may motivate further work. The algorithm runs in quasipolynomial time because they use limited enumeration.

💡 Research Summary

Dictionary learning, also known as sparse coding, seeks to recover an unknown dictionary matrix A ∈ ℝⁿˣᵐ and sparse coefficient vectors x ∈ ℝᵐ from observations y = Ax. While heuristic methods dominate practice, provable algorithms have only been available under very restrictive sparsity regimes—typically when each x is at most O(√n)‑sparse. Early breakthroughs (Spielman et al., 2012) handled the square case (m = n), and later works (Arora et al., 2013; Agarwal et al., 2013) extended to over‑complete dictionaries (m > n) but still required x to be O(√n)‑sparse and the dictionary to be incoherent (pairwise column inner products bounded by μ/√n).

The present paper asks whether one can provably learn dictionaries when the hidden vectors are much less sparse—specifically, when sparsity grows up to n / poly(log n). To achieve this, the authors introduce a new structural property of dictionaries called “individually recoverable features.” Intuitively, a feature (column of A) should be identifiable by looking only at the pixels it significantly influences, without being confusable with combinations of other features that occur with non‑negligible probability.

The authors formalize this intuition using two graph‑based assumptions on the bipartite support graph G_b formed by entries of A whose magnitude exceeds a threshold b.

Assumption 1 (Significant effect): Every column j has at least d entries with magnitude ≥ σ, i.e., each feature influences at least d pixels strongly.

Assumption 2 (Low pairwise intersection): In the graph G_τ where τ ≈ 1/ log n, the neighborhoods of any two columns intersect in at most d/10 entries (total weight ≤ dσ/10). A stronger version, Assumption 2′, bounds the intersection size for any pair by κ = O(d/ log² n).

For non‑negative dictionaries (all A_{ij} ≥ 0) the authors further normalize so that each pixel’s expected value is 1 and bound each entry by a constant Λ. Under these assumptions, they design an algorithm that proceeds in four stages:

1. Sample collection: Draw N = poly(n) observations y_i = A x_i where each x_i follows an independent ρ‑Bernoulli distribution (each coordinate is 1 with probability ρ).

2. Candidate generation: For each pixel i, enumerate columns j whose entry exceeds τ. Because of the low‑intersection assumption, the number of candidates per pixel is only poly(log n).

3. Pairwise verification: For each candidate pair (j, k), compute empirical covariances of the corresponding pixel sets. Since the probability that both features are simultaneously active is ρ², a small covariance indicates that the two columns act independently, confirming they are distinct features.

4. Clustering and reconstruction: Accepted columns are clustered to form the estimated dictionary. In the non‑negative case scaling is trivial; for general real‑valued dictionaries an additional variance‑control assumption (G3) ensures that many tiny entries do not dominate the pixel variance.

The main theoretical guarantees are:

Theorem 1 (Non‑negative case): Under Assumptions 1 and 2, if ρ = o(1/ log^{2.5} n), the algorithm runs in time n·O((Λ log² n)/σ⁴) using poly(n) samples and outputs a matrix ε‑equivalent to the true A (i.e., for a fresh random x, Ax and \hat A x differ entry‑wise by at most ε). With the stronger Assumption 2′ the output is n − C‑equivalent, requiring n^{4C+3} samples for a constant C depending on the parameters.

Theorem 2 (General real‑valued case): Under analogous assumptions G1, G2′, and G3, the same algorithm (with modest modifications) runs in n·O((Δ Λ log² n)/σ²) time, uses n^{4C+5}·m samples, and returns an n − C‑equivalent dictionary.

Both results allow sparsity ρ⁻¹ = Ω(n/ poly(log n)), dramatically extending the regime beyond √n. The runtime is quasi‑polynomial because the enumeration of candidate columns is limited to poly(log n) per pixel; this is analogous to the limited enumeration used in learning Gaussian mixtures.

The paper also discusses the practical relevance of the assumptions. Non‑negative dictionaries arise naturally in image processing (e.g., non‑negative matrix factorization) and the “large effect” condition often holds because filters tend to have localized strong responses. The low‑intersection condition is satisfied by many structured dictionaries (e.g., localized wavelet bases, convolutional filters) but not by dense random matrices, which explains why the algorithm does not apply to generic RIP matrices.

Limitations include the quasi‑polynomial runtime (still super‑polynomial for very large n) and the need for the structural graph assumptions, which may not hold for all real‑world dictionaries. The authors suggest future work on (i) empirical validation on natural image patches, (ii) relaxing the graph conditions to cover broader classes of dictionaries, and (iii) developing truly polynomial‑time algorithms perhaps by leveraging higher‑order statistics or more sophisticated combinatorial designs.

In summary, the paper makes a significant theoretical contribution by identifying a new “individually recoverable” property, proving that dictionaries satisfying this property can be learned provably when the hidden vectors are sparsity up to n / poly(log n), and providing concrete algorithms with rigorous error and sample complexity bounds. This bridges a major gap between the sparsity levels allowed by sparse recovery (which can handle O(n) sparsity) and those previously achievable for dictionary learning.