Rejoinder: Harold Jeffreyss Theory of Probability Revisited

We are grateful to all discussants of our re-visitation for their strong support in our enterprise and for their overall agreement with our perspective. Further discussions with them and other leading statisticians showed that the legacy of Theory of Probability is alive and lasting. [arXiv:0804.3173]

💡 Research Summary

The paper is a formal rejoinder to the extensive discussion that followed the authors’ revisitation of Harold Jeffreys’s Theory of Probability. It begins by expressing sincere gratitude to all discussants for their thoughtful critiques, constructive suggestions, and overall support. The authors emphasize that the dialogue has sharpened their arguments and helped clarify the enduring relevance of Jeffreys’s ideas in contemporary statistics.

The core of the rejoinder is a systematic response to the main points raised by the discussants. First, the authors reaffirm the philosophical foundation of Jeffreys’s prior: it is derived from the principle of information‑theoretic invariance, namely the minimisation of the expected Kullback–Leibler divergence or, equivalently, the maximisation of the determinant of the Fisher information matrix. This construction yields a prior that is “objective” in the sense that it does not depend on arbitrary parameterisations and provides a balanced weighting across the entire parameter space. The authors acknowledge concerns about potential subjectivity, but they argue that the information‑minimisation criterion is mathematically well‑defined and leads to a unique prior for a given model class.

Second, the paper presents a series of concrete examples that demonstrate the practical performance of Jeffreys priors. For the normal location‑scale family, the Jeffreys prior reproduces the familiar non‑informative prior proportional to 1/σ, and the resulting posterior mean coincides with the maximum‑likelihood estimator when the sample size is large. Similar consistency is shown for Bernoulli success probabilities (Beta(½,½) prior) and Poisson rates (Gamma(½,0) prior). In each case, the authors provide both analytical derivations and Monte‑Carlo simulations to illustrate that the posterior concentrates around the true parameter as data accumulate, confirming the asymptotic Bayes‑frequentist agreement that Jeffreys emphasised.

Third, the discussants questioned the applicability of Jeffreys priors to non‑regular or high‑dimensional models. In response, the authors introduce the concept of reference priors, a generalisation of Jeffreys’s construction that adapts the information‑minimisation principle to hierarchical, non‑linear, or multi‑parameter settings. They outline how reference priors are obtained by sequentially maximising the expected information for each parameter conditional on the others, thereby preserving the objective spirit while accommodating complex structures. This addresses the critique that Jeffreys’s original formulation is limited to simple, regular models.

Fourth, the rejoinder surveys contemporary domains where Jeffreys‑type priors have found renewed utility. In Bayesian neural networks, Jeffreys priors are employed to initialise weights in a scale‑invariant manner, improving training stability. In optimal experimental design, the Jeffreys information matrix serves as the canonical criterion for D‑optimality, guiding the selection of informative data points. Moreover, the authors cite recent work on Bayes factors and model selection, where Jeffreys priors provide a principled baseline for comparing nested models without inflating evidence through arbitrary prior choices.

Finally, the authors outline a forward‑looking research agenda. They propose developing efficient computational algorithms for constructing Jeffreys or reference priors in high‑dimensional spaces, investigating harmonisation techniques for combining priors across sub‑models, and integrating Jeffreys‑based inference with modern validation frameworks such as posterior predictive checks and cross‑validation. They also advocate for incorporating Jeffreys’s objective Bayesian philosophy into graduate curricula, arguing that a solid grounding in information‑theoretic priors equips the next generation of statisticians and data scientists with a robust, principled approach to uncertainty quantification.

In conclusion, the rejoinder reaffirms that Harold Jeffreys’s Theory of Probability remains a vibrant and influential component of statistical science. By addressing criticisms, showcasing empirical robustness, extending the methodology to complex models, and highlighting contemporary applications, the authors demonstrate that Jeffreys’s legacy is not merely historical but actively shapes modern Bayesian practice and will continue to do so in future research.