Fractal and Regular Geometry of Deep Neural Networks

We study the geometric properties of random neural networks by investigating the boundary volumes of their excursion sets for different activation functions, as the depth increases. More specifically, we show that, for activations which are not very regular (e.g., the Heaviside step function), the boundary volumes exhibit fractal behavior, with their Hausdorff dimension monotonically increasing with the depth. On the other hand, for activations which are more regular (e.g., ReLU, logistic and $\tanh$), as the depth increases, the expected boundary volumes can either converge to zero, remain constant or diverge exponentially, depending on a single spectral parameter which can be easily computed. Our theoretical results are confirmed in some numerical experiments based on Monte Carlo simulations.

💡 Research Summary

The paper investigates the geometric properties of random deep neural networks in the infinite‑width limit by treating them as isotropic Gaussian random fields on the sphere. The authors focus on the behavior of the boundary volume of excursion sets as the depth L increases, and they show that the activation function determines two fundamentally different regimes.

First, they introduce the Covariance Regularity Index (CRI), which quantifies the smoothness of the covariance kernel κ near the poles (t → 0). When κ admits an expansion κ(1−t)=p(t)+c t^β+o(t^β) with β∈(0,1), the CRI lies in (0,1) and the network belongs to the “fractal class”. This occurs for highly non‑regular activations such as the Heaviside step function. Using strong local nondeterminism, the authors derive the exact uniform modulus of continuity of the field, construct δ‑nets to bound the Hausdorff dimension of the graph, and apply the potential method to obtain matching lower bounds. They prove that the Hausdorff dimension of both the graph and the excursion boundary is non‑integer and increases monotonically with depth, converging to the ambient dimension d as L→∞. Hence deeper networks become “more fractal”.

Second, for smoother activations (ReLU, leaky‑ReLU, logistic, tanh, etc.) the CRI falls in (1,2]. The paper proves (Proposition 3.16) that any Gaussian field with CRI>1 is almost surely C¹, which allows the use of the classical Kac‑Rice formula. The expected boundary volume of an excursion set then depends only on the derivative of κ near 1. To connect CRI with the spectral decay of the angular power spectrum C_ℓ, the authors define a spectral index α via C_ℓ≈ℓ^{-(α+d)}. Proposition 3.4 shows that CRI = min(2, α/2) (except for the logarithmic case α=2). Consequently, three regimes emerge, mirroring the low‑disorder, sparse, and high‑disorder regimes identified in earlier work on deep Gaussian processes:

• Low‑disorder (α>2, CRI=2): the expected boundary volume decays to zero as depth grows.
• Sparse (α≈2, 1<CRI<2): the expected boundary volume remains constant with depth; ReLU falls here.
• High‑disorder (α<2, CRI∈(1,2)): the expected boundary volume grows exponentially with depth.

The theoretical results are corroborated by Monte‑Carlo simulations on S². For the Heaviside activation, visualizations show increasingly intricate excursion boundaries and a measured Hausdorff dimension that approaches 2 as L increases. For ReLU, logistic, and tanh, the simulated expected boundary volumes match the predicted zero, constant, or exponential behaviors depending on the activation’s spectral index.

Key contributions of the work include:

1. A precise link between the decay of the angular power spectrum (α) and the smoothness of the covariance kernel (CRI).
2. A novel fractal‑geometry analysis for networks with CRI<1, establishing exact Hausdorff dimension bounds via strong local nondeterminism.
3. A rigorous proof that CRI>1 guarantees almost‑sure C¹ regularity, enabling Kac‑Rice calculations of expected excursion boundary volumes.
4. A unified classification of deep random networks into fractal and Kac‑Rice classes, with explicit formulas for how depth and activation affect geometric complexity.

The paper opens a new perspective on neural‑network complexity by connecting depth, activation regularity, and geometric fractality. It suggests future directions such as studying how training (e.g., neural‑tangent kernel regimes) modifies the CRI, extending the analysis to non‑spherical domains, and exploring implications for generalization and expressivity.