Physics informed learning of orthogonal features with applications in solving partial differential equations

The random feature method (RFM) constructs approximation spaces by initializing features from generic distributions, which provides universal approximation properties to solve general partial differential equations. However, such standard initializations lack awareness of the underlying physical laws and geometry, which limits approximation. In this work, we propose the Physics-Driven Orthogonal Feature Method (PD-OFM), a framework for constructing feature representations that are explicitly tailored to both the differential operator and the computational domain by pretraining features using physics-informed objectives together with orthogonality regularization. This pretraining strategy yields nearly orthogonal feature bases. We provide both theoretical and empirical evidence that physics-informed pretraining improves the approximation capability of the learned feature space. When employed to solve Helmholtz, Poisson, wave, and Navier-Stokes equations, the proposed method achieves residual errors 2-3 orders of magnitude lower than those of comparable methods. Furthermore, the orthogonality regularization improves transferability, enabling pretrained features to generalize effectively across different source terms and domain geometries for the same PDE.

💡 Research Summary

The paper introduces the Physics‑Driven Orthogonal Feature Method (PD‑OFM), a novel framework that enhances the random feature method (RFM) for solving partial differential equations (PDEs) by incorporating physics‑informed pretraining and an orthogonality regularization term. Traditional RFM builds an approximation space from shallow neural networks whose weights are sampled from generic distributions; only the linear coefficients of the final layer are optimized via a least‑squares fit to the PDE residuals. While this approach enjoys universal approximation guarantees, the randomly initialized features are oblivious to the underlying differential operator and domain geometry, often leading to ill‑conditioned system matrices and poor approximation of high‑frequency or complex solution components.

PD‑OFM addresses these shortcomings in two complementary ways. First, it trains the hidden‑layer features themselves using a physics‑informed loss (L_PDE) that penalizes the residual of the governing equation and the mismatch of boundary conditions, much like a PINN but with the crucial difference that the loss is applied to the feature extractor rather than the full network. Second, it adds an orthogonality regularizer L_orth = ‖UᵀU – I‖², where U(x;θ) denotes the m‑dimensional feature vector produced by the penultimate layer. This term forces the feature matrix to be nearly orthogonal, which (i) reduces the condition number of the resulting linear system, (ii) encourages the features to resemble the eigenfunctions of the differential operator, and (iii) spreads the expressive power evenly across all features, avoiding the “few‑dominant‑feature” phenomenon observed with random bases.

The authors provide theoretical analysis showing that orthogonalization increases the effective rank of the feature space and yields a more uniform gradient distribution during training, thereby mitigating the risk of getting trapped in poor local minima. They also define a projection error metric that quantifies the distance between the span of learned features and the true eigenspace of the operator; experiments demonstrate that this error shrinks as orthogonality regularization is strengthened.

Empirical evaluation covers four representative PDE families: the Helmholtz equation, Poisson’s equation, the wave equation, and the incompressible Navier‑Stokes equations. In a 1‑D Poisson benchmark, the learned features closely match analytical sine eigenfunctions, and the approximation error decays much faster with the number of features than with random bases. For higher‑dimensional and nonlinear problems, PD‑OFM achieves residual errors that are two to three orders of magnitude lower than those obtained by standard RFM, while requiring fewer iterations of the least‑squares solver. Notably, the method exhibits strong transferability: once a feature set is pretrained for a given operator, it can be frozen and reused to solve the same PDE with different source terms or on altered geometries (e.g., non‑rectangular domains) with minimal loss of accuracy. This mirrors transfer learning practices in deep learning and opens the door to efficient parametric or multi‑query PDE solvers.

In summary, the main contributions are: (1) a physics‑informed pretraining scheme that demonstrably improves the approximation quality of feature spaces; (2) an orthogonality‑regularized loss that yields nearly orthogonal, operator‑adapted bases and dramatically improves numerical stability; (3) extensive experiments confirming superior accuracy, stability, and transferability compared with random feature methods and conventional PINNs. PD‑OFM thus offers a compelling hybrid between classical spectral methods (which rely on orthogonal bases) and modern machine‑learning‑based PDE solvers, providing a scalable, mesh‑free approach that can handle high‑dimensional, complex‑geometry problems with enhanced efficiency.