Linear Latent Force Models using Gaussian Processes

Purely data driven approaches for machine learning present difficulties when data is scarce relative to the complexity of the model or when the model is forced to extrapolate. On the other hand, purely mechanistic approaches need to identify and specify all the interactions in the problem at hand (which may not be feasible) and still leave the issue of how to parameterize the system. In this paper, we present a hybrid approach using Gaussian processes and differential equations to combine data driven modelling with a physical model of the system. We show how different, physically-inspired, kernel functions can be developed through sensible, simple, mechanistic assumptions about the underlying system. The versatility of our approach is illustrated with three case studies from motion capture, computational biology and geostatistics.

💡 Research Summary

The paper introduces a hybrid modeling framework called Latent Force Models (LFMs) that blends Gaussian process (GP) regression with linear differential equations to embed physical knowledge into data‑driven learning. The authors begin by contrasting pure data‑driven approaches, which struggle when data are scarce or when extrapolation is required, with purely mechanistic models, which can be overly complex and difficult to parameterize. LFMs occupy a middle ground: they retain the flexibility of GPs while incorporating a simple mechanistic prior expressed as a linear ordinary (or partial) differential equation driven by latent forces.

The core idea is to treat the unobserved driving inputs as latent functions u_q(t) that follow independent GP priors. The observable outputs y_d(t) are linked to these forces through a linear dynamical system. In its most general form the system is written as
  ¨Y M + ˙Y C + Y B = U S + E,
where M, C, and B are diagonal matrices of masses, dampings, and spring constants, S encodes sensitivities, and E is Gaussian noise. By solving the linear ODE (or PDE for spatio‑temporal cases) the output can be expressed as a convolution of the latent forces with the system’s impulse response. Because convolution with a linear operator preserves Gaussianity, the outputs are themselves GP‑distributed, and the resulting covariance (kernel) functions can be derived analytically by convolving the latent‑force kernel with the impulse response.

Two concrete instances are worked out in detail. The first‑order LFM (no mass term, unit damping) reduces to ˙Y + Y B = U S + E. Assuming a squared‑exponential kernel for each latent force, the authors derive a non‑stationary output kernel involving exponentials and error functions, which captures exponential decay and time‑lag effects. The cross‑covariance between outputs and latent forces is also derived, enabling exact posterior inference over the latent forces.

The second‑order LFM introduces masses, yielding ¨Y M + ˙Y C + Y B = U S + E. The impulse response now contains damped sinusoidal terms, so the resulting kernel reflects resonance, natural frequencies, and damping ratios. This formulation can model mechanical systems, RLC circuits, or any phenomenon governed by second‑order linear dynamics.

Inference proceeds by constructing the marginal likelihood of the observed outputs, which depends on the hyper‑parameters: length‑scales of the latent‑force kernels, the physical parameters (M, C, B, S), and noise variances. These are learned by maximizing the marginal likelihood using gradient‑based optimization. Since exact GP inference scales as O(N³), the authors discuss sparse approximations (e.g., inducing‑point methods such as FITC) to make the approach tractable for larger datasets.

Three case studies demonstrate the versatility of LFMs:

1. Human Motion Capture – A second‑order LFM models joint trajectories. The learned mass and damping parameters capture inertia and oscillatory behavior, outperforming PCA and standard GP baselines in reconstruction error.

2. Drosophila Gene Expression – A first‑order LFM is applied to time‑course gene expression data. The latent forces correspond to unobserved transcriptional regulators; the model uncovers biologically plausible regulatory dynamics and provides uncertainty‑aware predictions.

3. Swiss Jura Geostatistics – A spatio‑temporal LFM based on a diffusion PDE models the spread of heavy metals in soil. The derived non‑stationary kernel accounts for spatial diffusion and temporal decay, yielding more accurate predictions than ordinary Kriging.

The related‑work section situates LFMs among multi‑output GPs, semi‑parametric latent factor models, and Kalman‑filter approaches, emphasizing that LFMs uniquely embed mechanistic differential operators directly into kernel design, thereby improving interpretability and extrapolation capability.

In conclusion, the paper shows that linear latent force models provide a principled, analytically tractable way to fuse physical insight with flexible non‑parametric learning. While limited to linear dynamics, the framework can be extended via linearization, hierarchical composition, or non‑linear kernels, opening avenues for broader applications in robotics, systems biology, and environmental modeling.