Separable Dictionary Learning

Many techniques in computer vision, machine learning, and statistics rely on the fact that a signal of interest admits a sparse representation over some dictionary. Dictionaries are either available analytically, or can be learned from a suitable training set. While analytic dictionaries permit to capture the global structure of a signal and allow a fast implementation, learned dictionaries often perform better in applications as they are more adapted to the considered class of signals. In imagery, unfortunately, the numerical burden for (i) learning a dictionary and for (ii) employing the dictionary for reconstruction tasks only allows to deal with relatively small image patches that only capture local image information. The approach presented in this paper aims at overcoming these drawbacks by allowing a separable structure on the dictionary throughout the learning process. On the one hand, this permits larger patch-sizes for the learning phase, on the other hand, the dictionary is applied efficiently in reconstruction tasks. The learning procedure is based on optimizing over a product of spheres which updates the dictionary as a whole, thus enforces basic dictionary properties such as mutual coherence explicitly during the learning procedure. In the special case where no separable structure is enforced, our method competes with state-of-the-art dictionary learning methods like K-SVD.

💡 Research Summary

The paper introduces Separable Dictionary Learning (SeDiL), a novel framework that represents a dictionary as the Kronecker product of two smaller dictionaries, D = B ⊗ A. This structure dramatically reduces both memory consumption and computational cost, allowing the learning of dictionaries for much larger image patches than traditional methods such as K‑SVD, which are limited to small patches due to resource constraints. By enforcing unit‑norm columns on A and B, the dictionaries lie on a product of spheres manifold, enabling the use of Riemannian optimization techniques.

The objective function combines three terms: (i) a reconstruction error ‖AX_jBᵀ − S_j‖F², (ii) a sparsity‑promoting penalty g(X)=∑ln(1+ρ|x|²), and (iii) a mutual‑coherence regularizer r(A)+r(B), where r(·)=−∑{i<j}ln(1−|d_iᵀd_j|²). The latter is a smooth log‑barrier that discourages highly correlated atoms while remaining differentiable, which is crucial for gradient‑based optimization.

Optimization proceeds on the product manifold M = ℝ^{a×b×m} × S(h,a) × S(w,b) using a Riemannian conjugate‑gradient algorithm with a non‑monotone line search. Gradients are projected onto the tangent spaces of the spheres, and geodesics are computed analytically as great‑circle arcs, allowing efficient updates without costly re‑orthogonalization.

Theoretical analysis shows that the mutual coherence of the Kronecker product satisfies µ(B⊗A)=max{µ(A),µ(B)}; consequently, minimizing r(A)+r(B) also bounds the coherence of the full dictionary. Lemma 1 and the derived inequalities provide the mathematical foundation for this claim.

Experimental validation includes two main scenarios. First, on 8×8 image patches, SeDiL learns both separable and non‑separable dictionaries. The non‑separable version matches the performance of state‑of‑the‑art K‑SVD, while the separable version outperforms the analytic discrete cosine transform (DCT) by a noticeable margin in PSNR. Second, a face database with 64×64 images is used to train a separable dictionary, which is then applied to large‑hole inpainting. The results demonstrate that the learned dictionary captures global facial structure and can reconstruct missing regions convincingly, surpassing both analytic and non‑separable learned dictionaries in visual quality.

Overall, SeDiL offers a powerful combination of structural efficiency, explicit coherence control, and rigorous Riemannian optimization. It enables dictionary learning for high‑dimensional signals that were previously infeasible, and its modular Kronecker formulation suggests straightforward extensions to 3‑D volumetric data, video sequences, or any domain where multi‑dimensional tensor structures are natural.