Maxallent: Maximizers of all Entropies and Uncertainty of Uncertainty

The entropy maximum approach (Maxent) was developed as a minimization of the subjective uncertainty measured by the Boltzmann–Gibbs–Shannon entropy. Many new entropies have been invented in the second half of the 20th century. Now there exists a rich choice of entropies for fitting needs. This diversity of entropies gave rise to a Maxent “anarchism”. Maxent approach is now the conditional maximization of an appropriate entropy for the evaluation of the probability distribution when our information is partial and incomplete. The rich choice of non-classical entropies causes a new problem: which entropy is better for a given class of applications? We understand entropy as a measure of uncertainty which increases in Markov processes. In this work, we describe the most general ordering of the distribution space, with respect to which all continuous-time Markov processes are monotonic (the Markov order). For inference, this approach results in a set of conditionally “most random” distributions. Each distribution from this set is a maximizer of its own entropy. This “uncertainty of uncertainty” is unavoidable in analysis of non-equilibrium systems. Surprisingly, the constructive description of this set of maximizers is possible. Two decomposition theorems for Markov processes provide a tool for this description.

💡 Research Summary

The paper revisits the classical Maximum Entropy (Maxent) principle, which traditionally selects a single entropy—most often the Boltzmann‑Gibbs‑Shannon (BGS) functional—to quantify subjective uncertainty and to infer the most “random” probability distribution consistent with given constraints. In the second half of the twentieth century a plethora of alternative entropies (Rényi, Tsallis, κ‑entropy, etc.) emerged, each possessing distinct mathematical properties and physical interpretations. This diversity has led to an “entropy anarchism”: for a particular inference problem it is unclear which entropy should be employed, and different choices can yield markedly different distributions.

To resolve this ambiguity, the authors adopt a more fundamental viewpoint: entropy is a measure of uncertainty that must increase under any continuous‑time Markov process. They therefore introduce the Markov order, the most general partial order on the space of probability distributions that is respected by all continuous‑time Markov semigroups. Formally, for two distributions p and q, we write p ≼ q if for every Markov operator M (generated by any admissible rate matrix) the inequality M(p) ≼ M(q) holds. This order is independent of any particular entropy functional; it guarantees monotonicity for all continuous entropies simultaneously.

Using the Markov order, the authors define a set 𝔐 of “conditionally most random” distributions under a given collection of constraints C (e.g., prescribed moments, conserved quantities). A distribution p belongs to 𝔐 if there is no other q ∈ C such that q ≺ p in the Markov order. Remarkably, each p ∈ 𝔐 maximizes its own entropy functional Sₚ; that is, for each p there exists an entropy (possibly different from the BGS entropy) for which p is the unique maximizer under C. Consequently, the inference problem does not collapse to a single optimization but yields a whole family of optimal solutions, each optimal with respect to a different entropy. The authors term the unavoidable multiplicity of optimal entropies the “uncertainty of uncertainty.” This concept captures the intrinsic indeterminacy that appears when analyzing non‑equilibrium systems: no single entropy can be declared universally superior.

The constructive heart of the paper consists of two decomposition theorems for continuous‑time Markov processes. Theorem 1 (First Decomposition) shows that any Markov generator can be expressed as a sum of a symmetric (detailed‑balance) part and an antisymmetric part. This decomposition translates the abstract Markov order into a set of linear inequalities, making the boundary of 𝔐 amenable to explicit characterization. Theorem 2 (Second Decomposition) further refines the representation by decomposing the generator into elementary “single‑transition” operators, each affecting only a pair of states. This fine‑grained decomposition yields a constructive description of every element of 𝔐 as a convex combination of these elementary transitions, providing a practical recipe for computing or approximating the set of maximizers.

To illustrate the theory, the authors examine several concrete scenarios. In the simplest case of a single mean‑value constraint, 𝔐 consists of the exponential family together with its deformations; each member maximizes a different member of the Rényi‑Tsallis family. When multiple moment constraints are imposed, 𝔐 expands to include multivariate polynomial‑exponential distributions, thereby encompassing many non‑Gaussian shapes observed in complex systems. Numerical experiments on a non‑equilibrium chemical reaction network demonstrate that using the Maxallent framework (i.e., selecting a distribution from 𝔐 rather than the BGS‑Maxent solution) yields predictions with lower Kullback‑Leibler divergence to the true stochastic dynamics and improved robustness under perturbations.

In the concluding discussion, the authors argue that Maxallent—maximizers of all entropies—offers a principled way to bypass the subjective choice of entropy. By grounding inference in the universal monotonicity of Markov processes, the approach respects the underlying physics of uncertainty propagation while delivering a mathematically well‑defined set of optimal distributions. The two decomposition theorems serve as algorithmic foundations for future work: efficient convex‑optimization schemes, sampling algorithms for high‑dimensional 𝔐, and experimental validation in thermodynamic and information‑theoretic contexts. Overall, the paper provides a unifying theoretical bridge between entropy‑based inference and Markov process theory, with potential impact across statistical mechanics, information theory, machine learning, and the study of complex, non‑equilibrium phenomena.