Empirical Gaussian Processes

Gaussian processes (GPs) are powerful and widely used probabilistic regression models, but their effectiveness in practice is often limited by the choice of kernel function. This kernel function is typically handcrafted from a small set of standard functions, a process that requires expert knowledge, results in limited adaptivity to data, and imposes strong assumptions on the hypothesis space. We study Empirical GPs, a principled framework for constructing flexible, data-driven GP priors that overcome these limitations. Rather than relying on standard parametric kernels, we estimate the mean and covariance functions empirically from a corpus of historical observations, enabling the prior to reflect rich, non-trivial covariance structures present in the data. Theoretically, we show that the resulting model converges to the GP that is closest (in KL-divergence sense) to the real data generating process. Practically, we formulate the problem of learning the GP prior from independent datasets as likelihood estimation and derive an Expectation-Maximization algorithm with closed-form updates, allowing the model handle heterogeneous observation locations across datasets. We demonstrate that Empirical GPs achieve competitive performance on learning curve extrapolation and time series forecasting benchmarks.

💡 Research Summary

The paper tackles a fundamental limitation of Gaussian processes (GPs): the reliance on a handcrafted kernel function that must be chosen from a small set of standard, often stationary, kernels. Such kernels impose strong prior assumptions, require expert tuning, and struggle with extrapolation tasks like learning‑curve prediction or long‑range time‑series forecasting.
To overcome these issues, the authors introduce Empirical Gaussian Processes (Empirical GPs), a framework that learns the GP prior directly from a corpus of historical, independent datasets. Instead of fitting hyper‑parameters of a parametric kernel, the method estimates the prior mean m_S(x) and covariance k_S(x,x′) by simple sample statistics: the empirical average of the observed functions and the empirical outer‑product of their centered versions. Because k_S is a sum of outer products, it is guaranteed to be positive semi‑definite, thus a valid kernel by construction.
Theoretical contributions are twofold. First, under mild continuity and Dudley entropy conditions, the empirical GP converges almost surely to the true GP defined by the underlying stochastic process as the number of historical samples S → ∞ (Proposition 1). Second, the limiting GP is shown to be the best Gaussian approximation of the true data‑generating distribution in the Kullback‑Leibler (KL) sense (Proposition 2). These results provide a solid statistical justification: Empirical GPs asymptotically recover the optimal Gaussian surrogate of any complex stochastic process.
In practice, data are observed only at discrete input locations, which may differ across datasets. The authors address two regimes.

1. Dense observations: When each dataset provides a relatively dense sampling of its underlying function, a simple linear interpolant reconstructs a continuous surrogate. The empirical mean and covariance can then be computed directly, and a singular‑value‑decomposition (SVD) of the centered observation matrix yields a low‑rank representation that reduces the computational cost from O(S) to O(M), where M is the number of retained eigen‑observations.
2. Sparse, heterogeneous observations: For realistic settings where each task supplies only a few irregularly placed points, the paper introduces a latent reference grid Z of size M. A stationary base kernel k_base defines interpolation weights W_i = k_base(X_i, Z) k_base(Z, Z)^{-1} that map latent values u_i = f(Z) to the observed locations X_i. The generative model assumes u_i ∼ N(μ, Σ) and y_i | u_i ∼ N(W_i u_i, σ² I). An Expectation‑Maximization (EM) algorithm is derived with closed‑form updates: the E‑step computes the Gaussian posterior of each u_i given its observations, and the M‑step aggregates the posterior means and covariances across all S tasks to update μ and Σ. The algorithm’s dominant cost lies in the E‑step (O(S(N_i M² + N_i³))), but the Woodbury identity can reduce inversion costs when N_i > M.
   A critical practical issue is over‑confidence in extrapolation: far from the reference grid the interpolation weights vanish, causing the predictive variance to collapse to the observation noise floor. To mitigate this, the authors decompose the learned prior into a base parametric component (μ_base, k_base) and a residual component (δμ, δΣ) that captures deviations learned from data. For any new input x, the final prior is constructed as
    μ(x) = μ_base(x) + W_x δμ,
    k(x, x′) = k_base(x, x′) + W_x δΣ W_{x′}ᵀ,
   where W_x are the same interpolation weights defined on the base kernel. This scheme guarantees consistency (exact recovery at the reference points) and robust uncertainty far from the data, as the model gracefully reverts to the well‑behaved base kernel.
   Empirical evaluation focuses on two challenging domains.

• Learning‑curve extrapolation: Historical learning curves from many prior experiments exhibit a rapid initial improvement followed by saturation. Empirical GPs automatically capture this non‑stationary shape and consistently achieve lower root‑mean‑square error (RMSE) than a Transformer‑based baseline that was specifically engineered for curve extrapolation.
• Time‑series forecasting: Benchmarks include financial indices and atmospheric CO₂ concentrations. Compared against handcrafted‑kernel GPs, LSTMs, Temporal Convolutional Networks, and recent deep learning forecasters, Empirical GPs deliver superior predictive accuracy and more calibrated uncertainty, especially for long‑horizon forecasts where stationary kernels typically fail.
  The paper also demonstrates computational efficiency: the SVD‑based compression and the EM updates enable training on thousands of independent datasets within minutes, a speedup over existing meta‑learning GP approaches that require expensive gradient‑based hyper‑parameter optimization.
  In summary, the work proposes a non‑parametric, data‑driven prior learning paradigm for Gaussian processes. By directly estimating the prior mean and covariance from historical data, it sidesteps the need for expert‑crafted kernels, accommodates non‑stationarity, heteroscedasticity, and domain‑specific correlations, and provides theoretical guarantees of optimality in the KL sense. The EM‑based learning algorithm, together with the residual‑augmentation scheme, yields a practical and scalable solution that outperforms both classic GP kernels and modern deep learning models on extrapolation‑heavy tasks. This framework opens avenues for robust Bayesian optimization, meta‑learning across diverse domains, and large‑scale time‑series analysis where reliable uncertainty quantification is essential.