Bayesian quadrature is a probabilistic, model-based approach to numerical integration, the estimation of intractable integrals, or expectations. Although Bayesian quadrature was popularised already in the 1980s, no systematic and comprehensive treatment has been published. The purpose of this survey is to fill this gap. We review the mathematical foundations of Bayesian quadrature from different points of view; present a systematic taxonomy for classifying different Bayesian quadrature methods along the three axes of modelling, inference, and sampling; collect general theoretical guarantees; and provide a controlled numerical study that explores and illustrates the effect of different choices along the axes of the taxonomy. We also provide a realistic assessment of practical challenges and limitations to application of Bayesian quadrature methods and include an up-to-date and nearly exhaustive bibliography that covers not only machine learning and statistics literature but all areas of mathematics and engineering in which Bayesian quadrature or equivalent methods have seen use.
Numerical computation of integrals, often representing expectations or normalization constants, is a pervasive practical problem throughout machine learning, statistics, scientific computing, and engineering. In the form studied in this survey, numerical integration consists in approximating the definite integral
of a real-valued integrand function f with respect to a probability measure P on a set D ⊆ R d . The approximation typically takes the form of a quadrature rule N i=1 w i f (x i ), which is a weighted sum of integrand evaluations at some nodes x i ∈ D. Numerical integration can be done on arbitrary non-Euclidean domains and can also incorporate information other than point evaluations, generalisations that we discuss in Section 2.5. Over the past two centuries a vast number of numerical integration techniques have been developed from different starting points and for different types of integration problems. The most popular of these are classical Gaussian quadratures that are constructed using polynomial exactness criteria and admit an elegant mathematical theory (Gautschi, 2004); easy-to-use and widely applicable Monte Carlo integration based on random sampling that is, no doubt, the most popular approach to approximating an integral (Caflisch, 1998); and quasi-Monte Carlo methods that use low-discrepancy sequences and can compute high-dimensional integrals effectively (Dick et al., 2013). Other approaches include sparse grid methods, Clenshaw-Curtis quadrature, the trapezoidal rule, and the topic of this survey, Bayesian quadrature.
Although its history can be traced further back, Bayesian quadrature was not popularized until the late 1980s by Persi Diaconis and Anthony O’Hagan (Diaconis, 1988;O’Hagan, 1991). Since then, there has been a steady stream of interest and contributions from machine learning and statistics communities (e.g., Kennedy, 1998;Rasmussen and Ghahramani, 2002;Osborne et al., 2012a;Briol et al., 2019) and, more recently, from the point of view of numerical analysis and approximation theory (Jagadeeswaran and Hickernell, 2019;Kanagawa et al., 2020;Santin et al., 2022). Bayesian quadrature falls within probabilistic numerics (Hennig et al., 2022), its defining characteristic being the interpretation of numerical integration as a statistical inference problem to which the Bayesian paradigm and methods can be brought to bear.
In Bayesian quadrature, a stochastic process prior placed on the integrand is conditioned on the “data” consisting of integrand evaluations. The mean of the resulting posterior distribution provides a point estimate for the integral, while the spread of the posterior quantifies uncertainty (see Figure 1.1). That explicit prior information, such as smoothness or structural properties, is easy to encode both conceptually and in practice by selection of an appropriate prior for the integrand distinguishes Bayesian quadrature from other numerical integration techniques. Priors that are “correct” or “good” result in fast rates of convergence and reliable quantification of uncertainty (see Figure 1.2). Non-Bayesian integration methods too encode various types of prior in-formation, but typically in a non-systematic and implicit manner. For example, Gaussian quadratures implicitly assume that the integrand is well approximated by polynomials; the trapezoidal rule that a sum of trapezoids approximates the integral; and Monte Carlo encodes no assumptions whatsoever besides square-integrability. While quasi-Monte Carlo methods use node placement to encode certain assumptions, Bayesian quadrature can couple any nodes with any prior information, making it extremely versatile. That any nodes proposed in the vast literature on numerical integration can be used in Bayesian quadrature is one of its great advantages. Another distinguishing feature is the statistically principled uncertainty quantification that Bayesian quadrature provides: the spread of the posterior tells one how much to trust the integral estimate. Note that this uncertainty quantification arises from the prior and is wholly distinct in character from the sample variance of Monte Carlo. Much methodology and theory has been developed for Bayesian quadrature over the years (worthy mentions include Xi et al. 2018;Chai and Garnett 2019;and Kanagawa et al. 2020), and applications have ranged from computer graphics (Marques et al., 2015) and engineering (Kumar et al., 2008;Dang et al., 2021) to cardiac and tsunami modelling (Oates et al., 2017b;Li et al., 2023). Despite a wealth of such contributions, no comprehensive treatment of Bayesian quadrature as a whole has ever been published. Perhaps partly for this reason, confusion continues to plague the field. For example, “Bayesian quadrature” is often used to refer exclusively to a single method (or some collection of methods) to tackle integration problems of a very specific form. Although comparisons of Bayesian quadrature methods to other numerical integrati
This content is AI-processed based on open access ArXiv data.