Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike’’ spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.

💡 Research Summary

The paper “Depth, Not Data: An Analysis of Hessian Spectral Bifurcation” challenges the prevailing view that the well‑known bulk‑and‑spike structure of the Hessian spectrum in deep learning is primarily inherited from an imbalance in the data covariance matrix. Instead, the authors demonstrate that this spectral bifurcation can arise purely from the architecture of the network, specifically its depth, even when the data are perfectly balanced (i.e., whitened).

Problem Setting
The authors focus on deep linear networks of depth L ≥ 2, where the output is a product of weight matrices (F(x)=W_L\cdots W_1x). They assume a population loss based on mean‑squared error and consider a data distribution such that the input covariance (\Sigma_{xx}) is the identity and the cross‑covariance (\Sigma_{yx}) is a rank‑r orthogonal projection (i.e., (\Sigma_{yx}=U I_r V^\top)). This “perfectly balanced” setting eliminates any source of spectral imbalance from the data.

Key Assumptions

1. Sufficient Width & Whitened Input – The smallest of the input and output dimensions is not a bottleneck, and (\Sigma_{xx}=I).
2. Alignment with Initialization – The target mapping aligns with the singular vectors of the initial weight product.
3. Uniform Spectral Initialization (USI) – In a special case, all singular values of the initial product are equal to a scalar (\mu).

Analytical Tools
The analysis relies on two main technical ingredients:

• Balanced Initialization – A procedure that samples a matrix (A) and distributes its singular values equally across all layers, ensuring that each layer starts as a scaled isometry.
• Gauss‑Newton Decomposition – The Hessian (H) of the population loss is split into an outer‑product term (H_o) and a functional term (H_f). For MSE loss, this decomposition aligns with the classic Gauss‑Newton approximation.

Shared Spectral Structure (Lemma 3.4)
Under the balanced initialization and the whitening assumptions, all weight matrices share a common diagonal spectrum (\Sigma^{1/L}t = \operatorname{diag}(\lambda{1,t},\dots,\lambda_{d^*,t})) at any time (t). The dynamics of each eigenvalue obey a simple scalar recursion
\