Optimal information deletion and Bayes' theorem

In this same journal, Arnold Zellner published a seminal paper on Bayes’ theorem as an optimal information processing rule. This result led to the variational formulation of Bayes’ theorem, which is the central idea in generalized variational inference. Almost 40 years later, we revisit these ideas, but from the perspective of information deletion. We investigate rules which update a posterior distribution into an antedata distribution when a portion of data is removed. In such context, a rule which does not destroy or create information is called the optimal information deletion rule and we prove that it coincides with the traditional use of Bayes’ theorem.

💡 Research Summary

The paper revisits Arnold Zellner’s seminal 1988 work, which showed that Bayes’ theorem can be derived as the solution to a variational optimization problem that minimizes information loss measured by Shannon entropy. Building on this foundation, the authors introduce the concept of “information deletion”: given a full data set D that has been used to obtain a posterior distribution p(θ|D), they ask how to update this posterior when a subset D₂ of the data must be removed, leaving only D₁ = D \ D₂.

To answer this, they define an information deletion rule (IDR) analogous to Zellner’s information‑processing rule (IPR). The inputs to the IDR are the full‑data posterior p(θ|D) and the likelihood contribution of the data to be deleted, p(D₂|θ). The output is an “ante‑data” distribution that should represent the posterior conditioned only on the remaining data D₁. The authors construct an information‑loss functional
Δ_IDR = H(p̂(θ|D₁)) −