Convergence of Differential Entropies -- II

We show that under convergence in measure of probability density functions, differential entropy converges whenever the entropy integrands $f_n |\log f_n|$ are uniformly integrable and tight – a direct consequence of Vitali’s convergence theorem. We give an entropy-weighted Orlicz condition: $\sup_n \int f_n, Ψ(|\log f_n|) < \infty$ for a single superlinear~$Ψ$, strictly weaker than the fixed-$α$ condition of Godavarti and Hero (2004). We also disprove the Godavarti-Hero conjecture that $α> 1$ could be replaced by $α_n \downarrow 1$. We recover the sufficient conditions of Godavarti–Hero, Piera–Parada, and Ghourchian-Gohari-Amini as corollaries. On bounded domains, we prove that uniform integrability of the entropy integrands is both necessary and sufficient – a complete characterization of entropy convergence.

💡 Research Summary

The paper tackles a fundamental question in information theory: under what additional assumptions does convergence of probability density functions (pdfs) imply convergence of their differential entropies? While earlier works—Godavarti and Hero (2004), Piera and Parada (2009), and Ghourchian, Gohari, and Amini (2017)—provided various sufficient conditions (boundedness, moment constraints, fixed‑α integrability, bounded density ratios, total‑variation convergence), none identified a single underlying mechanism that unifies these results.

The authors introduce the entropy integrand sequence (g_n = f_n |\log f_n|). They prove that if ({g_n}) is uniformly integrable and tight (UI&T)—a condition comprising three parts: (A) a uniform (L^1) bound, (B) equi‑integrability (the usual UI condition), and (C) tail concentration on a set of finite measure—then Vitali’s convergence theorem (in its σ‑finite version) guarantees (g_n \to g) in (L^1), where (g = f |\log f|). Since differential entropy is simply (-\int g), this yields (H(f_n) \to H(f)). Lemma 5 formalizes this implication.

On bounded domains (\Omega\subset\mathbb R^d), condition (C) is automatic, so UI alone becomes both necessary and sufficient for entropy convergence (Theorem 7). The authors give a detailed proof that UI ⇒ entropy convergence (via Lemma 5) and conversely that entropy convergence forces UI (by splitting (g_n) into positive and negative parts and applying dominated convergence on each).

To provide a practical sufficient condition, the paper leverages the de la Vallée‑Poussin theorem. They show that the existence of a single convex, super‑linear function (\Psi) (i.e., (\Psi(t)/t\to\infty)) such that \