A Random Matrix--Theoretic Approach to Handling Singular Covariance Estimates

In many practical situations we would like to estimate the covariance matrix of a set of variables from an insufficient amount of data. More specifically, if we have a set of $N$ independent, identically distributed measurements of an $M$ dimensional random vector the maximum likelihood estimate is the sample covariance matrix. Here we consider the case where $N<M$ such that this estimate is singular and therefore fundamentally bad. We present a radically new approach to deal with this situation. Let $X$ be the $M\times N$ data matrix, where the columns are the $N$ independent realizations of the random vector with covariance matrix $\Sigma$. Without loss of generality, we can assume that the random variables have zero mean. We would like to estimate $\Sigma$ from $X$. Let $K$ be the classical sample covariance matrix. Fix a parameter $1\leq L\leq N$ and consider an ensemble of $L\times M$ random unitary matrices, ${\Phi}$, having Haar probability measure. Pre and post multiply $K$ by $\Phi$, and by the conjugate transpose of $\Phi$ respectively, to produce a non–singular $L\times L$ reduced dimension covariance estimate. A new estimate for $\Sigma$, denoted by $\mathrm{cov}_L(K)$, is obtained by a) projecting the reduced covariance estimate out (to $M\times M$) through pre and post multiplication by the conjugate transpose of $\Phi$, and by $\Phi$ respectively, and b) taking the expectation over the unitary ensemble. Another new estimate (this time for $\Sigma^{-1}$), $\mathrm{invcov}_L(K)$, is obtained by a) inverting the reduced covariance estimate, b) projecting the inverse out (to $M\times M$) through pre and post multiplication by the conjugate transpose of $\Phi$, and by $\Phi$ respectively, and c) taking the expectation over the unitary ensemble. We have a closed analytical expression for $\mathrm{invcov}_L(K)$ and $\mathrm{cov}_L(K)$ in terms of its eigenvalue decomposition.

💡 Research Summary

In many modern data‑analysis tasks the number of available samples N is far smaller than the dimensionality M of the random vector of interest. In this regime the maximum‑likelihood estimator of the covariance matrix, the sample covariance K = XX†/N, is rank‑deficient (rank ≤ N) and therefore singular. A singular covariance matrix cannot be inverted, which precludes its use in a wide range of algorithms that rely on the inverse covariance (e.g., linear discriminant analysis, Mahalanobis distance, Markowitz portfolio optimization). Traditional remedies such as adding a ridge term, shrinkage toward the identity, or performing principal‑component truncation mitigate the problem but do not fundamentally resolve the lack of information.

The authors propose a radically different approach based on random matrix theory and Haar‑distributed unitary transformations. Let X be the M × N data matrix whose columns are independent realizations of a zero‑mean random vector with true covariance Σ. Choose an integer L with 1 ≤ L ≤ N and draw a random unitary matrix Φ of size L × M from the Haar measure. The reduced‑dimension covariance estimate is defined as

 K̃ = Φ K Φ†,

which is an L × L matrix. Because L ≤ N, K̃ is almost surely full rank and thus invertible. Two new estimators are then constructed by projecting back to the original M‑dimensional space and averaging over the ensemble of Φ:

1. Covariance estimator

 cov_L(K) = E_Φ