On Empirical Spectral Distributions for Random Tensor Product Models

In statistics, assuming samples are independent is reasonable. However, this property can fail to hold for the features, a distinction that has led to several lines of work aiming to remove the latter assumption of independence present in the early literature, while preserving the original conclusions. Empirical spectral distributions of covariance matrices are key for understanding the data, and their almost sure convergence is oftentimes desirable. The random tensor product model, $X=(x_{i_1}x_{i_2}…x_{i_d})_{1 \leq i_1<…<i_d \leq n}$ for $x_1,x_2,\hspace{0.05cm}…\hspace{0.05cm},x_n$ i.i.d., introduced by the machine learning community, has a dependence structure for its features far from trivial and has been studied in recent years. When $x_1 \in \mathbb{R}, \mathbb{E}[x_1^4]<\infty, \frac{d}{n^{1/3}}=o(1),$ the empirical spectral distributions of the covariance matrices were proved to converge almost surely to Marchenko-Pastur laws in the random matrix theory regime. This work extends this result to the range $\frac{d}{n^{1/2}}=~o(1)$ when $x_1$ is symmetric with a subgaussian norm slowly growing in $n$ (the aforesaid range arises naturally, and the result failing when $\frac{d}{n^{1/2}} \to \infty$ appears to be a plausible claim) and shows that similarly to the case with independent features, the almost sure convergence holds under more general conditions on the covariance structure than the isotropic case. The latter result provides a means of deriving convergence for empirical spectral distributions of random matrices, applicable to other models as well so long as their entries exhibit a certain degree of concentration.

💡 Research Summary

The paper studies the empirical spectral distribution (ESD) of covariance matrices built from the random tensor‑product model, a construction that has become popular in modern machine learning. Given i.i.d. scalar variables (x_1,\dots,x_n), the model forms a matrix (Z\in\mathbb{R}^{N\times p}) whose columns are all distinct monomials of degree (d): each column is a vector (Z_0) with entries (x_{i_1}x_{i_2}\dots x_{i_d}) for (1\le i_1<\dots<i_d\le n). The sample covariance matrix is (S=\frac1pZZ^{\top}). Earlier work (Bryson et al., 2020) proved that, when (d/n^{1/3}=o(1)) and (\mathbb{E}