Infusing Experimental Reality into Complex Many-Body Hamiltonians: The Observable-Constrained Variational Framework (OCVF)

Deep learning potentials for complex many-body systems often face challenges of insufficient accuracy and a lack of physical realism. This paper proposes an “Observable-Constrained Variational Framework” (OCVF), a general top-down correction paradigm designed to infuse physical realism into theoretical “skeleton” models (H_o) by imposing constraints from macroscopic experimental observables (\mathfrak{O}_{\text{exp},s}). We theoretically derive OCVF as a numerically tractable extension of the “Constrained-Ensemble Variational Method” (CEVM), wherein a neural network (ΔH_θ) learns the correction functional required to match the experimental data. We apply OCVF to BaTiO3 (BTO) to validate the framework: a neural network potential trained on DFT data serves as H_o, and experimental PDF data at various temperatures are used as constraints (\mathfrak{O}{\text{exp},s}). The final model, H_o + ΔH_θ, successfully predicts the complete phase transition sequence accurately (s’, s \neq s’). Compared to the prior model, the accuracy of the Cubic-Tetragonal (C-T) phase transition temperature is improved by 95.8% , and the Orthorhombic-Rhombohedral (O-R) T_c accuracy is improved by 36.1%. Furthermore, the lattice structure accuracy in the Rhombohedral (R) phase is improved by 55.6%, validating the efficacy of the OCVF framework in calibrating theoretical models via observational constraints.

💡 Research Summary

The paper introduces the Observable‑Constrained Variational Framework (OCVF), a top‑down methodology designed to correct systematic biases in deep‑learning interatomic potentials that are trained solely on density‑functional theory (DFT) data. Traditional “bottom‑up” machine‑learning potentials inherit the intrinsic errors of the underlying DFT functional (e.g., PBE), leading to inaccurate predictions of macroscopic properties such as phase‑transition temperatures and low‑temperature structural stability. To overcome this, OCVF treats a DFT‑based potential as a “skeleton” Hamiltonian H₀ and augments it with a learnable correction term ΔHθ, where θ denotes the parameters of a neural network.

The theoretical foundation is the Constrained‑Ensemble Variational Method (CEVM), which seeks a corrected probability distribution that (i) reproduces a set of experimental macroscopic observables {𝒪_exp,s} (e.g., temperature‑dependent pair‑distribution functions) and (ii) remains as close as possible to the prior distribution defined by H₀, measured by the Kullback‑Leibler (KL) divergence. Solving the CEVM variational problem yields a correction of the form ΔH = −kBT ∑_s λ_s Ô_s, where Ô_s are microscopic operators associated with the observables and λ_s are Lagrange multipliers. Because a limited operator basis cannot capture the full complexity of many‑body errors, the authors replace the rigid linear ansatz with a flexible, non‑linear neural‑network ansatz ΔHθ, leveraging the universal approximation theorem.

Training proceeds by defining a loss function L(θ) = ∑_s D_s(⟨Ô_s⟩_θ, 𝒪_exp,s), where ⟨Ô_s⟩_θ denotes the ensemble average of operator Ô_s computed from molecular dynamics (MD) simulations driven by the total Hamiltonian H_c = H₀ + ΔHθ. The MD engine is made fully differentiable (e.g., a Nose‑Hoover‑Chain NPT integrator) so that gradients can be back‑propagated through the simulation. The total gradient ∂L/∂θ decomposes into three components: (1) the observational gradient ∂D_s/∂O_sim, quantifying the discrepancy between simulated and experimental PDFs; (2) the physical gradient ∂F_s/∂H_c, measuring the sensitivity of the observable to infinitesimal changes in the potential energy surface, obtained via the adjoint sensitivity method; and (3) the model gradient ∂ΔHθ/∂θ, supplied by standard autograd.

The framework is validated on barium titanate (BaTiO₃), a prototypical ferroelectric perovskite with a well‑known cubic‑tetragonal‑orthorhombic‑rhombohedral (C‑T‑O‑R) transition sequence. A DFT‑PBE trained neural‑network potential (the prior H₀) reproduces the transition temperatures qualitatively but fails dramatically at low temperatures, yielding non‑physical structures. By incorporating experimental PDF data at multiple temperatures as constraints, OCVF learns ΔHθ that aligns the simulated PDFs with experiment. After training, the cubic‑tetragonal transition temperature error is reduced by 95.8 %, the orthorhombic‑rhombohedral transition error improves by 36.1 %, and the lattice‑parameter accuracy in the rhombohedral phase increases by 55.6 %. These improvements demonstrate that experimental macroscopic constraints can effectively correct the systematic deficiencies of DFT‑based potentials.

From a methodological perspective, OCVF differs from existing approaches such as variational force‑matching (which matches forces/energies from DFT) and differentiable trajectory reweighting (which avoids differentiating the MD solver). Instead, OCVF directly differentiates the full MD trajectory, ensuring that the corrected Hamiltonian influences the dynamics in a physically consistent way. The use of the minimum‑relative‑entropy principle guarantees that the correction introduces the smallest possible information loss relative to the prior, while the neural‑network correction provides the flexibility needed for complex many‑body systems.

Implementation challenges include the high computational cost of differentiable NPT simulations and the need for large ensembles to obtain stable observable averages. The authors mitigate these issues by employing a GPU‑accelerated DimeNet++ architecture for ΔHθ, using the adjoint method to reduce memory overhead, and selectively applying constraints to focus learning on the most informative temperature points.

Future directions suggested by the authors involve expanding the set of experimental constraints (e.g., dielectric response, specific heat), applying OCVF to multi‑component alloys and heterostructures, and exploring transfer‑learning strategies to reuse learned corrections across chemically similar systems. The framework also opens the possibility of integrating other sources of macroscopic data, such as neutron scattering or Raman spectra, to further refine interatomic potentials.

In summary, OCVF provides a theoretically grounded, practically implementable route to embed experimental reality into machine‑learned many‑body Hamiltonians. By marrying the rigor of constrained variational principles with the expressive power of deep neural networks, it achieves substantial improvements in predictive accuracy for complex phase behavior, establishing a new paradigm for the calibration of interatomic potentials against macroscopic observations.