Optimal Bias-variance Tradeoff in Matrix and Tensor Estimation

We study matrix and tensor denoising when the underlying signal is \textbf{not} necessarily low-rank. In the tensor setting, we observe [ Y = X^\ast + Z \in \mathbb{R}^{p_1 \times p_2 \times p_3}, ] where $X^\ast$ is an unknown signal tensor and $Z$ is a noise tensor. We propose a one-step variant of the higher-order SVD (HOSVD) estimator, denoted $\widetilde X$, and show that, uniformly over any user-specified Tucker ranks $(r_1,r_2,r_3)$, with high probability, [ |\widetilde X - X^\ast|{\mathrm F}^2 = O\Big( κ^2\Big{r_1r_2r_3 + \sum{k=1}^3 p_k r_k\Big} + ξ_{(r_1,r_2,r_3)}^2 \Big). ] Here, $ξ_{(r_1,r_2,r_3)}$ is the best achievable Tucker rank-$(r_1,r_2,r_3)$ approximation error of $X^\ast$ (bias), $κ^2$ quantifies the noise level, and $κ^2{r_1r_2r_3+\sum_{k=1}^3 p_k r_k}$ is the variance term scaling with the effective degrees of freedom of $\widetilde X$. This yields a rank-adaptive bias-variance tradeoff: increasing $(r_1,r_2,r_3)$ decreases the bias $ξ_{(r_1,r_2,r_3)}$ while increasing variance. In the matrix setting, we show that truncated SVD achieves an analogous bias-variance tradeoff for arbitrary signal matrices. Notably, our matrix result requires \textbf{no} assumptions on the signal matrix, such as finite rank or spectral gaps. Finally, we complement our upper bounds with matching information-theoretic lower bounds, showing that the resulting bias-variance tradeoff is minimax optimal up to universal constants in both the matrix and tensor settings.

💡 Research Summary

The paper addresses the fundamental problem of denoising matrices and third‑order tensors without assuming that the underlying signal is exactly low‑rank. In the matrix setting, the authors consider the observation model Y = X* + Z where Z is a sub‑Gaussian noise matrix. They analyze the rank‑r truncated singular value decomposition (SVD) estimator Y(r) and prove a deterministic bound that holds uniformly for any target rank r:

‖Y(r) – X*‖_F ≤ (2+√2)(√r‖Z‖ + ξ(r)),

where ξ(r)=‖X* – X*(r)‖_F is the optimal rank‑r approximation error (the bias) and √r‖Z‖ captures the variance incurred by estimating r singular components. By invoking concentration results for sub‑Gaussian matrices, they convert the deterministic bound into a high‑probability risk bound

‖Y(r) – X*‖_F^2 = O(κ^2 r (m+n) + ξ(r)^2).

A matching information‑theoretic lower bound is established, showing that the above upper bound is minimax optimal up to universal constants. The analysis requires no structural assumptions on X*: its rank may be full, and no spectral gap is needed. The authors also illustrate the result in covariance matrix estimation, demonstrating the practical relevance of the bias‑variance trade‑off.

In the tensor setting, the observation model is Y = X* + Z with X* ∈ ℝ^{p1×p2×p3} and i.i.d. sub‑Gaussian noise entries. The paper proposes a one‑step variant of the higher‑order SVD (HOSVD). Given user‑specified Tucker ranks (r1,r2,r3), the algorithm computes the leading r_k singular vectors of each mode‑k unfolding, then refines them by projecting onto the Kronecker product of the other two mode subspaces, and finally reconstructs the estimator \tilde X by multilinear multiplication. Under a singular‑value gap condition that scales with the noise level κ and the dimensions (essentially a signal‑to‑noise ratio requirement), they prove with high probability

‖\tilde X – X*‖F ≤ C₃ √{ κ^2 (∑{k=1}^3 p_k r_k + r1 r2 r3) } + ξ(r1,r2,r3),

where ξ(r1,r2,r3) is the best achievable Tucker‑rank-(r1,r2,r3) approximation error (bias). The first term reflects the effective degrees of freedom of the estimator (variance). A relative error version is also derived, showing that when the ranks are modest (r_k ≤ √p_min) the normalized error is bounded by a constant times (κ^2 Σ p_k σ_{r_k}^2 + ξ^2)/‖X*‖_F^2. A matching minimax lower bound is proved, establishing that the rate κ^2(∑ p_k r_k + r1 r2 r3) + ξ^2 is optimal up to constants.

The paper includes remarks on computational and storage benefits: the truncated SVD reduces matrix multiplication from O(mn) to O(mr+nr) and storage from O(mn) to O(mr+nr); similarly, the Tucker decomposition reduces tensor operations from O(p1p2p3) to O(r1p1 + r2p2 + r3p3 + r1r2r3). Numerical experiments on synthetic data and a real 3‑D brain MRI dataset confirm the theoretical bias‑variance trade‑off, showing that the empirical mean‑squared error follows the predicted curve as the target ranks vary.

Overall, the contributions are: (1) a rank‑adaptive bias‑variance decomposition for matrix denoising that holds without any low‑rank or spectral gap assumptions; (2) an analogous result for tensors using a simple one‑step HOSVD, again without exact Tucker‑low‑rank assumptions; (3) matching minimax lower bounds proving optimality; (4) new perturbation bounds for matrices and tensors that may be of independent interest; and (5) practical algorithms with clear computational advantages, validated on real data. This work bridges the gap between theory and practice for high‑dimensional denoising when signals are only approximately low‑rank.