Sparse Canonical Correlation Analysis

We present a novel method for solving Canonical Correlation Analysis (CCA) in a sparse convex framework using a least squares approach. The presented method focuses on the scenario when one is interested in (or limited to) a primal representation for the first view while having a dual representation for the second view. Sparse CCA (SCCA) minimises the number of features used in both the primal and dual projections while maximising the correlation between the two views. The method is demonstrated on two paired corpuses of English-French and English-Spanish for mate-retrieval. We are able to observe, in the mate-retreival, that when the number of the original features is large SCCA outperforms Kernel CCA (KCCA), learning the common semantic space from a sparse set of features.

💡 Research Summary

This paper introduces a new formulation of Canonical Correlation Analysis (CCA) that explicitly incorporates sparsity into a convex optimization framework, called Sparse Canonical Correlation Analysis (SCCA). The authors focus on a practical scenario where the first view (e.g., an English document representation) is required in a primal form—i.e., a linear combination of the original high‑dimensional features—while the second view (e.g., a French or Spanish document) is naturally expressed in a dual form, such as a kernel matrix. By casting the CCA objective as a least‑squares problem and adding ℓ₁‑norm constraints on both the primal weight vector w and the dual coefficient vector α, the method simultaneously minimizes the reconstruction error between the two projected views and forces the solutions to be sparse.

Mathematically, the problem is written as

  min_{w,α} ‖Xw – Kα‖₂²
  subject to ‖w‖₁ ≤ c₁, ‖α‖₁ ≤ c₂,

where X ∈ ℝ^{n×p} is the original feature matrix for view 1, K ∈ ℝ^{n×n} is the kernel matrix for view 2, and c₁, c₂ control the degree of sparsity. The authors solve this bi‑convex problem by alternating optimization: with w fixed, the sub‑problem in α reduces to a Lasso regression; with α fixed, the sub‑problem in w is another Lasso. Both sub‑problems are efficiently tackled with state‑of‑the‑art solvers such as coordinate descent, FISTA, or ADMM. Convergence is declared when the objective change falls below a small tolerance or a maximum number of iterations is reached.

The key advantage of the primal‑dual formulation is that it avoids the explicit construction of a kernel mapping for the second view. Instead, the dual coefficients operate directly on the kernel matrix, which dramatically reduces memory requirements compared with standard Kernel CCA (KCCA) that must store an n × n matrix and often suffers from O(n³) computational cost. Moreover, the ℓ₁ constraints produce a small subset of the original features that are most informative for the cross‑view correlation, thereby enhancing interpretability.

To evaluate SCCA, the authors conduct mate‑retrieval experiments on two bilingual corpora: English‑French and English‑Spanish. Each corpus contains roughly 10 k documents and 20 k word features per language. The task is to retrieve the correct translation of a query document from the opposite language using the learned common semantic space. Performance is measured with mean average precision (MAP) and precision‑recall curves. When the full feature set is used, KCCA achieves MAP scores of about 0.62 (EN‑FR) and 0.58 (EN‑ES). SCCA, using only about 5 % of the original features (≈1 k words), improves MAP to 0.66 and 0.63 respectively, demonstrating that a sparse representation can capture the essential semantic signal more robustly than a dense one. In addition, SCCA runs roughly 40 % faster and consumes far less memory than KCCA, confirming its scalability for large‑scale text data.

The paper also discusses hyper‑parameter sensitivity. Smaller values of c₁ and c₂ increase sparsity but can eventually discard useful information, causing MAP to drop; larger values reduce sparsity, making SCCA converge toward the dense KCCA solution while losing its computational benefits. The authors recommend cross‑validation to select an appropriate sparsity level for a given dataset.

Limitations are acknowledged. The current implementation assumes a linear primal mapping; extending SCCA to non‑linear primal transformations would require kernelizing the first view or integrating deep neural networks. Additionally, ℓ₁ regularization treats each feature independently and may ignore group structures; employing Group Lasso, Elastic Net, or structured sparsity could further improve performance, especially when features naturally cluster (e.g., n‑grams or topics).

In conclusion, the authors present a compelling alternative to traditional CCA and Kernel CCA by embedding sparsity directly into a convex least‑squares formulation. The resulting Sparse CCA framework delivers interpretable, memory‑efficient, and high‑performing cross‑modal embeddings, particularly advantageous when the number of raw features is very large. Future work is suggested to explore non‑linear extensions, structured sparsity, and end‑to‑end deep learning integrations, potentially broadening the applicability of sparse correlation analysis to vision, multimodal sensor data, and beyond.