Hessian Spectral Analysis at Foundation Model Scale

Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful spectral analysis of the true Hessian is tractable at frontier scale. Using shard-local finite-difference Hessian vector products compatible with Fully Sharded Data Parallelism, we perform stochastic Lanczos quadrature on open-source language models with up to 100B parameters, producing the first large-scale spectral density estimates beyond the sub-10B regime. We characterize the numerical behavior of this pipeline, including finite-difference bias, floating-point noise amplification, and their effect on Krylov stability in fp32 and bf16, and derive practical operating regimes that are validated empirically. We further provide end-to-end runtime and memory scaling laws, showing that full-operator spectral probing incurs only a modest constant-factor overhead over first-order training. Crucially, direct access to the Hessian reveals that widely used block-diagonal curvature approximations can fail catastrophically, exhibiting order-one relative error and poor directional alignment even in mid-scale LLMs. Together, our results demonstrate that foundation-model Hessian spectra are both computable and qualitatively misrepresented by prevailing approximations, opening the door to principled curvature-based analysis at scale.

💡 Research Summary

The paper tackles a long‑standing obstacle in deep‑learning research: obtaining accurate Hessian spectra for foundation‑scale models. While prior work either limited itself to modestly sized networks or relied on strong structural approximations (e.g., block‑diagonal, K‑FAC), this study demonstrates that exact spectral analysis of the true Hessian is feasible even for models with up to 100 billion parameters.

Key technical contribution – a shard‑local finite‑difference Hessian‑vector product (HVP) that works natively with Fully Sharded Data Parallel (FSDP) training. By perturbing each GPU’s local parameter shard by ±ε v, performing two forward‑backward passes on the same data slice, and scaling the difference of the resulting gradients, the method yields an unbiased estimate of H v without any full‑parameter gathers. The additional communication cost is limited to the scalar reductions already required for Lanczos orthogonalisation, keeping the overhead comparable to two extra gradient passes.

Numerical analysis – The authors derive a rigorous error bound for the finite‑difference estimator that incorporates machine epsilon (ε_mach). They show the optimal perturbation magnitude ε* scales as (ε_mach ‖∇L‖ ‖∇³_v L‖)^{1/3}, leading to an error of order ε_mach^{2/3}. In practice, ε*≈10⁻³–10⁻² works well for FP32, while ε*≈10⁻¹ is appropriate for BF16, matching the theoretical predictions.

Lanczos integration – Using the stochastic Lanczos quadrature (SLQ) framework, the paper proves that when the HVPs are approximated by the finite‑difference scheme, the tridiagonal matrix produced by m Lanczos steps deviates from the exact one by a perturbation ΔT_m whose expected spectral norm is bounded by a term proportional to ‖H‖₂, the fourth‑order derivative norm, and a factor η that captures the delocalisation of Krylov vectors. This analysis explains the emergence of “ghost” eigenvalues when orthogonalisation is insufficient and quantifies their RMS splitting.

Systems cost model – A single HVP costs 2 · T_grad + T_vec, where T_grad is the wall‑clock time of a standard distributed gradient evaluation and T_vec accounts for local AXPY operations and a scalar all‑reduce. One Lanczos iteration therefore costs T_HvP plus a small number of scalar reductions. The total runtime of SLQ with s probe vectors and m Lanczos steps is approximately s · m · T_lanczos, which the authors empirically validate to be only 1.3–1.7× the cost of a regular training epoch on 8‑GPU nodes for the 100 B model. Memory overhead is modest (<5 % of the model footprint).

Empirical findings – The authors apply the pipeline to open‑source language models of 7 B, 30 B, and 100 B parameters. The estimated spectral densities exhibit the familiar bulk‑plus‑outlier shape, with the leading eigenvalue growing dramatically as model size increases, indicating increasingly sharp curvature directions. Crucially, when comparing the exact Hessian to widely used block‑diagonal approximations (K‑FAC, EK‑FAC, etc.), they observe order‑one relative Frobenius error and cosine similarities below 0.3, demonstrating that these approximations can be catastrophically inaccurate even for mid‑scale models.

Implications – By showing that true Hessian spectra are computable at foundation‑model scale with only a modest constant‑factor overhead, the work opens the door to a host of second‑order applications: curvature‑aware learning‑rate schedules, preconditioned optimizers, real‑time monitoring of loss‑landscape sharpness, and exact influence‑function calculations for data governance. The paper also calls into question the reliability of existing curvature approximations and suggests that future research should pivot toward methods that directly exploit the full Hessian, now that the computational barrier has been lowered.

In summary, the study delivers a practical, theoretically grounded, and scalable framework for Hessian spectral analysis on the largest models to date, establishing that second‑order information is no longer a theoretical curiosity but a tractable tool for the next generation of AI systems.