Low-Rank Tensor Decompositions for the Theory of Neural Networks

The groundbreaking performance of deep neural networks (NNs) promoted a surge of interest in providing a mathematical basis to deep learning theory. Low-rank tensor decompositions are specially befitting for this task due to their close connection to NNs and their rich theoretical results. Different tensor decompositions have strong uniqueness guarantees, which allow for a direct interpretation of their factors, and polynomial time algorithms have been proposed to compute them. Through the connections between tensors and NNs, such results supported many important advances in the theory of NNs. In this review, we show how low-rank tensor methods–which have been a core tool in the signal processing and machine learning communities–play a fundamental role in theoretically explaining different aspects of the performance of deep NNs, including their expressivity, algorithmic learnability and computational hardness, generalization, and identifiability. Our goal is to give an accessible overview of existing approaches (developed by different communities, ranging from computer science to mathematics) in a coherent and unified way, and to open a broader perspective on the use of low-rank tensor decompositions for the theory of deep NNs.

💡 Research Summary

This review paper surveys how low‑rank tensor decompositions have become a central mathematical tool for understanding deep neural networks (NNs). It begins by motivating the need for a rigorous theory of deep learning, noting that despite impressive empirical successes, many fundamental questions about expressivity, learnability, generalization, and identifiability remain open. The authors argue that low‑rank tensors—long used in signal processing, machine learning, and applied mathematics—provide a natural bridge because the weight tensors of many NN architectures can be interpreted as tensor objects with rich algebraic structure.

The paper first introduces the main tensor formats: Canonical Polyadic Decomposition (CPD), Tucker decomposition, and tree‑based tensor networks such as Hierarchical Tucker (HT) and Tensor Train (TT). CPD is highlighted for its strong uniqueness (identifiability) properties under mild Kruskal‑type conditions, which allow a direct interpretation of the factors as latent components of a network. Tucker‑type formats are more flexible and enable dramatic compression, but they lack generic uniqueness. The authors discuss generic properties of tensor decompositions—generic rank, typical ranks, and generic identifiability—using tools from algebraic geometry. These results give precise bounds on how many rank‑one components a tensor of a given size can have, and they directly translate into expressive power bounds for neural architectures.

Computational aspects are examined next. While exact CPD is NP‑hard in general, polynomial‑time algorithms exist for many regimes (e.g., when factor matrices satisfy certain incoherence or orthogonality conditions). Local optimization methods such as alternating least squares work well in practice, and recent perturbation analyses guarantee stability under noise. In contrast, Tucker, TT, and HT formats admit stable, linear‑algebra‑based algorithms (e.g., higher‑order SVD) that are well‑conditioned and scalable.

The core of the review is organized around four thematic applications.

1. Weight Compression – Representing weight tensors of MLPs, CNNs, Transformers, and RNNs in low‑rank formats reduces memory and compute. The paper surveys works on static low‑rank factorization, dynamic low‑rank updates (e.g., LoRA), and theoretical analyses of how compression affects convergence and implicit bias.
2. Expressivity and Approximation – By mapping certain NN families (notably Sum‑Product Networks) to CPD, researchers have quantified the advantage of depth over width, derived separation results, and linked approximation rates to classical function spaces such as Besov spaces. Algebraic‑geometric techniques have also been used to study identifiability of polynomial and linear networks.
3. Learning with Derivatives – The authors discuss how the Jacobian and higher‑order derivatives of a network can be expressed as low‑rank tensors, enabling polynomial‑time learning algorithms for 2‑ and 3‑layer networks via method‑of‑moments or power‑iteration schemes. These results provide guarantees on sample complexity, generalization, and parameter recovery even with flexible activation functions.
4. Emerging Problems – Tensor methods have been applied to generative models parameterized by polynomial NNs, hidden Markov models, restricted Boltzmann machines, reinforcement‑learning value‑function parameterizations, and mixtures of linear classifiers. In each case, tensor rank bounds give insight into learnability and computational hardness.

The review concludes by emphasizing that low‑rank tensor theory offers a unified framework linking four pillars of deep‑learning theory: expressivity, algorithmic learnability, generalization, and identifiability. Open challenges include extending uniqueness results to higher‑order nonlinear tensor formats, integrating more general activation functions, and designing scalable algorithms for real‑time compression of massive models. The authors anticipate that continued cross‑disciplinary work between algebraic geometry, numerical linear algebra, and deep‑learning will further illuminate the theoretical foundations of modern neural networks.