Visualizing the loss landscapes of physics-informed neural networks

Training a neural network requires navigating a high-dimensional, non-convex loss surface to find parameters that minimize this loss. In many ways, it is surprising that optimizers such as stochastic gradient descent and ADAM can reliably locate minima which perform well on both the training and test data. To understand the success of training, a “loss landscape” community has emerged to study the geometry of the loss function and the dynamics of optimization, often using visualization techniques. However, these loss landscape studies have mostly been limited to machine learning for image classification. In the newer field of physics-informed machine learning, little work has been conducted to visualize the landscapes of losses defined not by regression to large data sets, but by differential operators acting on state fields discretized by neural networks. In this work, we provide a comprehensive review of the loss landscape literature, as well as a discussion of the few existing physics-informed works which investigate the loss landscape. We then use a number of the techniques we survey to empirically investigate the landscapes defined by the Deep Ritz and squared residual forms of the physics loss function. We find that the loss landscapes of physics-informed neural networks have many of the same properties as the data-driven classification problems studied in the literature. Unexpectedly, we find that the two formulations of the physics loss often give rise to similar landscapes, which appear smooth, well-conditioned, and convex in the vicinity of the solution. The purpose of this work is to introduce the loss landscape perspective to the scientific machine learning community, compare the Deep Ritz and the strong form losses, and to challenge prevailing intuitions about the complexity of the loss landscapes of physics-informed networks.

💡 Research Summary

The paper “Visualizing the loss landscapes of physics‑informed neural networks” investigates how the loss surfaces of physics‑informed machine learning (PIML) models behave, and whether they share the same geometric properties that have been documented for data‑driven image‑classification networks. The authors begin with a comprehensive review of the loss‑landscape literature, dividing it into three methodological pillars: (i) mathematical analyses that prove general properties such as the absence of bad local minima in over‑parameterized networks, hierarchical minima, and permutation invariance; (ii) statistical‑physics perspectives that model stochastic gradient descent (SGD) as a stochastic differential equation, linking loss to an energy function and revealing concepts such as flat minima, phase transitions, and Gibbs‑type steady‑states; and (iii) empirical studies that rely on visual tools—random direction slices, Hessian eigen‑spectra, Goldilocks zones, mode connectivity, monotonic linear interpolation, intrinsic dimensionality, and low‑dimensional subspace training.

Having set this context, the authors turn to physics‑informed neural networks (PINNs) and the Deep Ritz method (DRM). PINNs typically minimize a squared residual of the governing PDE (the “strong form”), possibly augmented with boundary‑condition penalties. DRM, by contrast, minimizes the variational energy (the “weak form”), which reduces the order of differentiation and naturally incorporates Neumann conditions. Although the two objectives are mathematically distinct, both aim at the same physical solution.

The experimental part of the paper focuses on two benchmark problems: (1) a one‑dimensional linear elliptic (Poisson) equation, and (2) a two‑dimensional Neo‑Hookean hyperelasticity problem. For each problem the authors train identical neural‑network architectures using both the strong‑form residual loss and the DRM energy loss, and then apply a battery of loss‑landscape diagnostics that have become standard in the image‑classification community.

Key findings include:

1. Smooth, well‑conditioned landscapes – Cross‑sections of the loss along random directions produce smooth contour plots without sharp spikes. Hessian analyses reveal that, near the minima, the spectrum is dominated by positive eigenvalues; negative curvature is essentially absent. This indicates strong local convexity, suggesting that second‑order optimizers (Newton, L‑BFGS) could be effective for PINNs.

2. Similarity between DRM and strong‑form losses – Despite differing formulations, the two loss surfaces exhibit almost identical geometry: the same Goldilocks zone (a radius range where the Hessian is most isotropic and training is fastest), comparable intrinsic dimensionalities, and identical mode‑connectivity paths. Consequently, the choice between DRM and residual loss appears to be a matter of implementation convenience rather than a driver of optimization difficulty.

3. Mode connectivity and monotonic linear interpolation – Solutions obtained from different random seeds or optimizers (Adam vs. SGD) can be connected by low‑loss curves (Bezier or spline paths) that never increase the loss. Moreover, linear interpolation between the initial and final parameters yields a monotonically decreasing loss, confirming the “MLI” phenomenon observed in classification networks.

4. Goldilocks zone and intrinsic dimensionality – The authors identify a narrow band of parameter norms where the Hessian eigenvalues are most uniform and training converges fastest. Random subspace experiments show that successful training can be confined to a subspace whose dimensionality is orders of magnitude smaller than the full parameter count, echoing findings from the data‑driven literature.

5. Absence of bad local minima – Across a range of learning rates, batch sizes, and acceleration schemes, the experiments never encounter a local minimum that is both low‑loss and poorly generalizing. This aligns with theoretical results on over‑parameterized networks and suggests that the physics‑informed loss itself contributes to a benign landscape.

The paper concludes that physics‑informed loss landscapes share many of the favorable properties previously documented for image‑classification networks, while also offering additional structure (variational convexity, lower intrinsic dimensionality) that can be exploited. The authors argue that these insights open the door to applying second‑order optimization, dimensionality‑reduction, and advanced visualization tools to large‑scale scientific computing problems. Future work is suggested in extending the analysis to more complex, multi‑physics PDEs, three‑dimensional domains, and automated detection of Goldilocks zones for adaptive training schedules.