Extension of Sparse Randomized Kaczmarz Algorithm for Multiple Measurement Vectors

The Kaczmarz algorithm is popular for iteratively solving an overdetermined system of linear equations. The traditional Kaczmarz algorithm can approximate the solution in few sweeps through the equations but a randomized version of the Kaczmarz algorithm was shown to converge exponentially and independent of number of equations. Recently an algorithm for finding sparse solution to a linear system of equations has been proposed based on weighted randomized Kaczmarz algorithm. These algorithms solves single measurement vector problem; however there are applications were multiple-measurements are available. In this work, the objective is to solve a multiple measurement vector problem with common sparse support by modifying the randomized Kaczmarz algorithm. We have also modeled the problem of face recognition from video as the multiple measurement vector problem and solved using our proposed technique. We have compared the proposed algorithm with state-of-art spectral projected gradient algorithm for multiple measurement vectors on both real and synthetic datasets. The Monte Carlo simulations confirms that our proposed algorithm have better recovery and convergence rate than the MMV version of spectral projected gradient algorithm under fairness constraints.

💡 Research Summary

The paper addresses the problem of solving an under‑determined linear system with multiple measurement vectors (MMV), B = AX, where the unknown matrix X is assumed to be row‑sparse (i.e., all columns share a common support). While several sparse recovery techniques exist for single measurement vectors (SMV)—including ℓ₁‑regularized methods (LASSO, SPGL1) and greedy algorithms (OMP, CoSaMP)—the recent Sparse Randomized Kaczmarz (SRK) algorithm offers a fast, greedy‑like approach for SMV but cannot directly enforce a common support across multiple right‑hand sides.

To fill this gap, the authors propose SRK‑MMV, a modification of SRK that operates on the MMV setting. The algorithm introduces two key changes: (1) Support selection is based on the ℓ₂‑norm of each row of the current estimate X, selecting the rows with the largest norms as the tentative support set S. This ensures that the support is common to all columns. (2) The projection step is performed for each measurement vector separately but using the same randomly chosen row a_i and the same weight vector w_j. For each column ℓ, the update is
 x^{(ℓ)} ← x^{(ℓ)} + (b_i^{(ℓ)} – ⟨w_j⊙a_i, x^{(ℓ)}⟩) (w_j⊙a_i) / ‖w_j⊙a_i‖₂²,
and the collection of updated columns forms the new X. Row selection follows the same probability distribution as in the original Randomized Kaczmarz (RK): P(i) = ‖a_i‖₂² / ‖A‖_F², which biases the algorithm toward rows with larger energy and accelerates convergence.

The algorithm requires an estimate of the sparsity level ˆK. Experiments on synthetic data (m=500, n=100, L=5, true sparsity K∈{10,20,30,40}) show that the relative root‑mean‑square error is minimized when ˆK is roughly twice the true K for small K, and slightly larger than K for larger K. When the number of sweeps (full passes over all rows) is varied, SRK‑MMV reaches a relative error of 10⁻³ within 5–10 sweeps, whereas the competing Spectral Projected Gradient MMV (SPG‑MMV) under the same computational budget stalls around 10⁻².

A practical application to video‑based face recognition is also presented. Each subject’s training video is treated as a dictionary V, and a test video (multiple frames) yields a measurement matrix ˆv_test. The MMV formulation ˆv_test = V ˆα is solved with SRK‑MMV, producing a row‑sparse coefficient matrix ˆα. Classification proceeds by computing class‑wise residuals ‖ˆv_test – V_i ˆα_i‖₂ and selecting the class with the smallest residual. On real video datasets, SRK‑MMV achieves 3–5 % higher recognition accuracy than SPG‑MMV and converges roughly 30 % faster.

The authors acknowledge several limitations: (i) dependence on an accurate initial sparsity estimate; (ii) lack of rigorous convergence or recovery guarantees—the weight and support updates are heuristic; (iii) the need to compute ‖A‖_F for the row‑selection probabilities, which may be non‑trivial when A is implicit (e.g., Fourier or wavelet operators). Nevertheless, the method is memory‑efficient (O(n) storage) and each iteration consists only of vector‑row inner products, making it attractive for real‑time or embedded implementations.

In conclusion, SRK‑MMV extends the fast, greedy nature of the SRK algorithm to the MMV setting, delivering superior empirical recovery rates and convergence speed compared to state‑of‑the‑art SPG‑MMV under fair computational constraints. Future work is suggested on automatic sparsity estimation, theoretical analysis of convergence, and integration with large‑scale implicit operators to broaden its applicability.