Cross-Validation for Unsupervised Learning

Cross-validation (CV) is a popular method for model-selection. Unfortunately, it is not immediately obvious how to apply CV to unsupervised or exploratory contexts. This thesis discusses some extensions of cross-validation to unsupervised learning, specifically focusing on the problem of choosing how many principal components to keep. We introduce the latent factor model, define an objective criterion, and show how CV can be used to estimate the intrinsic dimensionality of a data set. Through both simulation and theory, we demonstrate that cross-validation is a valuable tool for unsupervised learning.

💡 Research Summary

The paper tackles a long‑standing challenge in unsupervised learning: how to choose model complexity when no explicit loss function or ground‑truth labels are available. While cross‑validation (CV) is a standard tool for model selection in supervised contexts, its direct application to tasks such as principal component analysis (PCA) is not straightforward. The authors propose a principled framework that extends CV to unsupervised settings by embedding the problem in a latent‑factor model.

In the latent‑factor formulation, an observed data matrix (X\in\mathbb{R}^{n\times p}) is approximated as (X\approx ZL^{\top}), where (Z) (size (n\times d)) contains the low‑dimensional scores and (L) (size (p\times d)) holds the loadings. The dimensionality (d) is the quantity to be selected. For a given (d), the authors fit the model on a training split, obtain (\hat{Z}) and (\hat{L}), and then compute the reconstruction error on a held‑out validation split: \