Thinning, Entropy and the Law of Thin Numbers

Renyi’s “thinning” operation on a discrete random variable is a natural discrete analog of the scaling operation for continuous random variables. The properties of thinning are investigated in an information-theoretic context, especially in connection with information-theoretic inequalities related to Poisson approximation results. The classical Binomial-to-Poisson convergence (sometimes referred to as the “law of small numbers” is seen to be a special case of a thinning limit theorem for convolutions of discrete distributions. A rate of convergence is provided for this limit, and nonasymptotic bounds are also established. This development parallels, in part, the development of Gaussian inequalities leading to the information-theoretic version of the central limit theorem. In particular, a “thinning Markov chain” is introduced, and it is shown to play a role analogous to that of the Ornstein-Uhlenbeck process in connection to the entropy power inequality.

💡 Research Summary

The paper revisits Rényi’s thinning operation—a discrete analogue of continuous scaling—and places it squarely within an information‑theoretic framework. Thinning is defined as the random deletion of each count in a discrete random variable X with independent Bernoulli(p) trials, producing a new variable Xₚ whose mean and variance are both multiplied by p. The authors first establish basic algebraic properties: thinning commutes with convolution, preserves the Poisson family (Poisson(λ) thinned by p becomes Poisson(pλ)), and can be interpreted as a Markov transition.

The core information‑theoretic results are two monotonicity theorems. The “Thinning Entropy Increase Theorem” proves that H(Xₚ) ≥ H(X) for 0 ≤ p ≤ 1, mirroring the entropy‑increase property of continuous scaling. The “Thinning KL‑Decrease Theorem” shows that the Kullback‑Leibler divergence to a Poisson target never increases under thinning: D(P_X‖Poisson(λ)) ≥ D(P_{Xₚ}‖Poisson(pλ)). These theorems provide a clean explanation of why thinning drives arbitrary discrete distributions toward Poisson law.

Building on this, the authors formulate a general “Thinning Limit Theorem”. For any sequence of i.i.d. discrete variables whose sum is convolved n times, if each summand is thinned by pₙ with n pₙ → λ and pₙ → 0, then the thinned sum converges in distribution to Poisson(λ). This subsumes the classical Binomial‑to‑Poisson “law of small numbers” as a special case. Importantly, the paper supplies explicit convergence rates: total variation distance and relative entropy decay at O(pₙ) (or O(1/√n) under mild moment conditions). Non‑asymptotic bounds are derived via a “filling function” that tightly controls Poisson tail probabilities, making the results applicable to finite‑sample scenarios.

A novel contribution is the introduction of the “thinning Markov chain”. In continuous time, the chain evolves by (1 – Δt)‑thinning followed by the addition of an independent Poisson(λΔt) increment. The Poisson(λ) distribution is the unique stationary distribution, and the chain’s entropy power N(t)=exp{2H(X_t)/k} (k being the dimension) is shown to increase monotonically, exactly as in the Ornstein‑Uhlenbeck (OU) process for Gaussian variables. Leveraging this dynamics, the authors prove an “discrete Entropy Power Inequality (EPI)”: for independent thinning chains X_t and Y_t, N(X_t+Y_t) ≥ N(X_t)+N(Y_t). This discrete EPI parallels the classic continuous‑variable EPI and provides a powerful tool for bounding information loss in additive discrete systems.

The paper concludes with several potential applications. In data compression, thinning can be used to adjust sparsity while preserving or increasing entropy, suggesting new variable‑length or sparse‑coding schemes. In network traffic modeling, packet arrivals can be thinned to justify Poisson approximations, improving queue‑ing analyses. In sparse signal recovery, thinning followed by Poisson‑based reconstruction may yield robustness against noise. The authors also outline future directions: multivariate thinning, hybrid continuous‑discrete models, and algorithmic exploitation of thinning‑induced monotonicity.

Overall, the work bridges discrete probability and information theory, offering a unified perspective on Poisson approximation, entropy dynamics, and functional inequalities. By mirroring the Gaussian central limit theorem’s information‑theoretic proof structure, it establishes thinning as the natural discrete counterpart, opening avenues for both theoretical advances and practical algorithm design.