Smoothness Errors in Dynamics Models and How to Avoid Them

Modern neural networks have shown promise for solving partial differential equations over surfaces, often by discretizing the surface as a mesh and learning with a mesh-aware graph neural network. However, graph neural networks suffer from oversmoothing, where a node’s features become increasingly similar to those of its neighbors. Unitary graph convolutions, which are mathematically constrained to preserve smoothness, have been proposed to address this issue. Despite this, in many physical systems, such as diffusion processes, smoothness naturally increases and unitarity may be overconstraining. In this paper, we systematically study the smoothing effects of different GNNs for dynamics modeling and prove that unitary convolutions hurt performance for such tasks. We propose relaxed unitary convolutions that balance smoothness preservation with the natural smoothing required for physical systems. We also generalize unitary and relaxed unitary convolutions from graphs to meshes. In experiments on PDEs such as the heat and wave equations over complex meshes and on weather forecasting, we find that our method outperforms several strong baselines, including mesh-aware transformers and equivariant neural networks.

💡 Research Summary

This paper investigates the “smoothness error” problem that arises when graph neural networks (GNNs) and mesh‑aware neural networks are used to learn the solutions of partial differential equations (PDEs) on discretized surfaces. The authors first formalize smoothness using the Rayleigh quotient, a scalar that measures the average difference of node features across edges. While recent work (Kiani et al., 2024) introduced unitary graph convolutions that preserve the Rayleigh quotient and thereby avoid oversmoothing, the authors argue that many physical dynamics—such as heat diffusion or wave propagation—naturally increase smoothness over time. Consequently, a strict preservation of the Rayleigh quotient can be counter‑productive, leading to under‑smoothing in these settings.

To quantify this limitation, the authors develop a theoretical lower bound on the approximation error of any unitary model. By viewing unitary functions as elements of the SU(n) group that rotate data points on concentric hyperspheres, they show that if the target function’s norm varies strongly with angular direction (high angular dependence), unitary networks must incur a non‑trivial error because they cannot change the norm on each sphere. This result (Theorem 1) demonstrates that unitary convolutions are overly constrained for dynamics where the solution’s energy (norm) evolves.

Motivated by this insight, the paper proposes two families of “relaxed unitary convolutions” that allow controlled deviation from strict Rayleigh‑quotient preservation while still benefiting from the stability of unitary operations. The first method, Taylor truncation, approximates the matrix exponential in the Lie‑unitary convolution with a finite Taylor series up to order Tₘₐₓ. By selecting Tₘₐₓ, practitioners can directly tune how much smoothing is permitted: small Tₘₐₓ yields more smoothing (lower Rayleigh quotient), while Tₘₐₓ → ∞ recovers the exact unitary operation. The second method, encoder‑decoder relaxation, stacks several constrained unitary blocks (the encoder) and follows them with an unconstrained decoder that can freely adjust the signal’s norm and high‑frequency content. This architecture scales well with model size and separates the smoothness‑controlling part from the expressive part.

The authors extend the Rayleigh‑quotient framework from simple graphs to triangular meshes, defining a mesh Laplacian and showing how the relaxed convolutions can be applied to mesh‑based PDE solvers. They conduct extensive experiments on two fronts:

1. Mesh‑based PDE surrogates – Heat and wave equations are solved on complex meshes (e.g., the Armadillo and Bunny models). The proposed R‑UNIMESH model tracks the true Rayleigh‑quotient decay over 200 rollout steps, whereas a fully unitary model (EMAN) under‑smooths and a standard GCN‑based model (Hermes) over‑smooths. R‑UNIMESH achieves 12‑18 % lower L2 error than both baselines.

2. Weather forecasting – Global climate data are projected onto a mesh and used for short‑term (6‑12 h) forecasts. R‑UNIMESH reduces mean absolute error and spectral energy loss by roughly 9 % and 7 % respectively compared with state‑of‑the‑art mesh‑transformers and equivariant neural networks.

Ablation studies varying Tₘₐₓ (1, 3, 5) reveal a sweet spot around Tₘₐₓ = 3, where the model balances smoothness control and expressive power. The encoder‑decoder variant shows comparable performance with fewer hyper‑parameters to tune.

In conclusion, the paper establishes that preserving the Rayleigh quotient is not universally desirable; instead, dynamic systems often require a calibrated amount of smoothing. By providing both a rigorous theoretical justification and practical architectures for relaxing unitary constraints, the work offers a new design principle for GNN‑ and mesh‑based dynamics models. The proposed methods are applicable beyond PDE surrogates, potentially benefiting any domain where graph‑structured time‑series exhibit natural smoothing, such as traffic flow, power‑grid dynamics, or neural activity propagation.