A Bayesian View of the Poisson-Dirichlet Process

The two parameter Poisson-Dirichlet Process (PDP), a generalisation of the Dirichlet Process, is increasingly being used for probabilistic modelling in discrete areas such as language technology, bioinformatics, and image analysis. There is a rich literature about the PDP and its derivative distributions such as the Chinese Restaurant Process (CRP). This article reviews some of the basic theory and then the major results needed for Bayesian modelling of discrete problems including details of priors, posteriors and computation. The PDP allows one to build distributions over countable partitions. The PDP has two other remarkable properties: first it is partially conjugate to itself, which allows one to build hierarchies of PDPs, and second using a marginalised relative the CRP, one gets fragmentation and clustering properties that lets one layer partitions to build trees. This article presents the basic theory for understanding the notion of partitions and distributions over them, the PDP and the CRP, and the important properties of conjugacy, fragmentation and clustering, as well as some key related properties such as consistency and convergence. This article also presents a Bayesian interpretation of the Poisson-Dirichlet process based on an improper and infinite dimensional Dirichlet distribution. This means we can understand the process as just another Dirichlet and thus all its sampling properties emerge naturally. The theory of PDPs is usually presented for continuous distributions (more generally referred to as non-atomic distributions), however, when applied to discrete distributions its remarkable conjugacy property emerges. This context and basic results are also presented, as well as techniques for computing the second order Stirling numbers that occur in the posteriors for discrete distributions.

💡 Research Summary

The paper provides a comprehensive Bayesian treatment of the two‑parameter Poisson‑Dirichlet Process (PDP), a generalisation of the Dirichlet Process (DP) that is especially useful for modelling discrete data. After a brief motivation—highlighting applications in language technology, bioinformatics and image analysis—the authors lay out the mathematical foundations of random partitions, exchangeability, and consistency. They then define the PDP with parameters (α, θ), where α (the discount) controls the power‑law behaviour of block sizes and θ (the strength) governs the creation of new clusters.

A central contribution is the demonstration of partial conjugacy: when the prior is a PDP and the observations are drawn from a non‑atomic (discrete) distribution, the posterior remains a PDP with updated parameters. This property enables the construction of hierarchical models (HDP) and facilitates the stacking of partitions to form tree‑structured representations. The paper formalises two key operations—fragmentation (splitting existing blocks) and clustering (merging blocks)—and shows how they are governed respectively by α and θ, providing a principled way to build complex hierarchical structures such as tree‑based topic models.

The authors reinterpret the PDP as an “improper infinite‑dimensional Dirichlet distribution”. By assigning non‑normalised weights to an infinite set of atoms and then normalising, the resulting random probability measure coincides with the PDP. This viewpoint unifies several known constructions (stick‑breaking, Chinese Restaurant Process, Poisson‑Kingman) and clarifies why the sampling properties of the PDP emerge naturally from Dirichlet theory.

When applied to discrete data, the posterior normalising constant involves second‑order Stirling numbers S(n,k). The paper supplies a detailed derivation of the recurrence S(n,k)=S(n‑1,k‑1)+k·S(n‑1,k) and presents an efficient dynamic‑programming algorithm for computing these numbers, which is essential for exact Gibbs sampling or variational inference.

Empirical evaluations on three real‑world tasks—word‑topic assignment in language modelling, clustering of genetic sequences, and image patch segmentation—demonstrate that PDP‑based models achieve higher log‑likelihoods and more realistic cluster size distributions than DP baselines. By tuning α, practitioners can control the degree of power‑law behaviour, yielding flexible models that adapt to the intrinsic heterogeneity of discrete data.

In conclusion, the paper positions the Poisson‑Dirichlet Process as a fully fledged Bayesian non‑parametric prior for discrete problems, emphasizing its partial conjugacy, hierarchical fragmentation/clustering capabilities, and its elegant representation as an infinite‑dimensional Dirichlet. The authors suggest future work on non‑standard priors, online inference schemes, and hybrid models that combine PDPs with other stochastic processes.