Exact Non-Parametric Bayesian Inference on Infinite Trees

Given i.i.d. data from an unknown distribution, we consider the problem of predicting future items. An adaptive way to estimate the probability density is to recursively subdivide the domain to an appropriate data-dependent granularity. A Bayesian would assign a data-independent prior probability to “subdivide”, which leads to a prior over infinite(ly many) trees. We derive an exact, fast, and simple inference algorithm for such a prior, for the data evidence, the predictive distribution, the effective model dimension, moments, and other quantities. We prove asymptotic convergence and consistency results, and illustrate the behavior of our model on some prototypical functions.

💡 Research Summary

The paper tackles the classic problem of density estimation and predictive inference from i.i.d. observations by introducing a fully Bayesian, non‑parametric model based on an infinite binary tree. The authors place a data‑independent prior on whether a node in the tree should split, using a Beta‑Bernoulli construction, while the leaf nodes are assigned a uniform distribution over the corresponding sub‑region of the input space. This yields a prior over a countably infinite set of possible tree structures, each representing a different adaptive partition of the domain.

A major technical contribution is the derivation of exact, closed‑form recursion formulas for the marginal likelihood (evidence), the posterior distribution over trees, and the predictive density. For a node (v) containing (n_v) data points, the evidence can be written as a weighted sum of two terms: one corresponding to the event that the node splits (involving a Beta function of the counts in the left and right child) and one corresponding to the event that the node stops (involving the normalizing constant of the uniform leaf). Because the recursion only depends on the counts in the children, the entire evidence can be computed in linear time with respect to the number of observations using dynamic programming. The posterior split probability (\mathbb{E}