Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimension

Deep neural networks have attained remarkable success across diverse classification tasks. Recent empirical studies have shown that deep networks learn features that are linearly separable across classes. However, these findings often lack rigorous justifications, even under relatively simple settings. In this work, we address this gap by examining the linear separation capabilities of shallow nonlinear networks. Specifically, inspired by the low intrinsic dimensionality of image data, we model inputs as a union of low-dimensional subspaces (UoS) and demonstrate that a single nonlinear layer can transform such data into linearly separable sets. Theoretically, we show that this transformation occurs with high probability when using random weights and quadratic activations. Notably, we prove this can be achieved when the network width scales polynomially with the intrinsic dimension of the data rather than the ambient dimension. Experimental results corroborate these theoretical findings and demonstrate that similar linear separation properties hold in practical scenarios beyond our analytical scope. This work bridges the gap between empirical observations and theoretical understanding of the separation capacity of nonlinear networks, offering deeper insights into model interpretability and generalization.

💡 Research Summary

This paper addresses a fundamental question in deep learning: why and how shallow nonlinear layers can transform data that are not linearly separable into representations that are. The authors focus on data that lie on a union of low‑dimensional subspaces (UoS), a model motivated by empirical observations that image classes often occupy low‑intrinsic‑dimensional manifolds within a high‑dimensional ambient space.

The theoretical contribution is a rigorous analysis of a single‑layer network of the form f_W(x)=σ(Wx), where σ is an entry‑wise quadratic activation (σ(z)=z²) and the weight matrix W∈ℝ^{D×d} has i.i.d. standard Gaussian entries. Under the assumption that there are exactly two subspaces (K=2) of equal dimension r, with all principal angles strictly positive (i.e., the subspaces intersect only at the origin), the authors prove that if the width D grows polynomially with the intrinsic dimension r, then with high probability the transformed point sets f_W(S₁) and f_W(S₂) become linearly separable. More precisely, they show that for D = O(r^c) (for some constant c depending on the desired confidence), there exists a separating hyperplane v∈ℝ^D such that vᵀf_W(x)>0 for all x∈S₁{0} and vᵀf_W(x)<0 for all x∈S₂{0}. The failure probability decays exponentially in D·θ_min², where θ_min is the smallest principal angle between the subspaces.

The proof proceeds in three main steps. First, the quadratic activation lifts each subspace into a higher‑order feature space where points are expressed as outer products of the original coordinates, effectively turning linear subspaces into quadratic manifolds. Second, the random Gaussian projection W spreads these manifolds across many independent directions, ensuring that the inner products between points from different subspaces become small in expectation. Third, concentration inequalities (matrix Bernstein) are used to bound deviations from the expectation, guaranteeing a positive margin between the two transformed sets with high probability.

The authors extend the result conceptually to more than two subspaces (K>2), showing that the required width scales as O(K·poly(r)). Although the formal proof for K>2 is omitted, the same geometric intuition applies.

Empirically, the paper validates the theory on synthetic data and several real image datasets (MNIST, Fashion‑MNIST, CIFAR‑10). Synthetic experiments vary the intrinsic dimension r and the network width D, revealing a sharp phase transition: once D exceeds a threshold proportional to r², linear separability (measured by a linear probe) jumps from chance level to near‑perfect accuracy. For real datasets, singular‑value analysis confirms that each class’s data matrix is dominated by a small fraction of singular values, supporting the UoS assumption. When the same shallow network (with quadratic or ReLU activations) is applied, the learned features become linearly separable, and the width required follows the same scaling with the estimated intrinsic dimension. Moreover, random weights already achieve separability at sufficient width, while trained weights can do so with smaller widths, suggesting that over‑parameterization and random feature models capture much of the phenomenon observed in trained deep networks.

The paper situates its contributions relative to prior work. Earlier studies on random ReLU networks (DGJS22, GMS22) showed linear separability but required widths exponential in the ambient dimension d, a gap the current work bridges by focusing on intrinsic dimension r. Connections to deep linear networks, neural tangent kernel (NTK), and neural collapse are discussed, highlighting how the present analysis offers a first step toward understanding why early layers in deep models often produce linearly separable representations.

Limitations include the reliance on equal‑dimensional subspaces, the focus on quadratic activation for the formal proof (though experiments suggest ReLU works similarly), and the lack of a full multi‑class (K>2) proof. Future directions proposed are extending the theory to heterogeneous subspace dimensions, noisy data, deeper architectures, and providing rigorous bounds for other activation functions.

In summary, the paper provides a clear theoretical mechanism—random Gaussian projection combined with a quadratic nonlinearity—that explains how a shallow network can separate data lying on low‑dimensional subspaces, with a width requirement that scales polynomially with the intrinsic data dimension rather than the ambient dimension. This insight helps reconcile empirical observations of early‑layer feature separability with rigorous mathematical understanding, and it points toward more efficient network designs that exploit intrinsic dimensionality.