Description length of canonical and microcanonical models

💡 Research Summary

The paper investigates the relationship between canonical (soft‑constraint) and micro‑canonical (hard‑constraint) maximum‑entropy models from the perspective of the Minimum Description Length (MDL) principle, focusing on binary matrix data that arise in many domains such as bipartite networks and time‑series. Both model families are derived by maximizing Shannon entropy under a set of constraints c(G). In the canonical formulation the constraints are satisfied only on average, leading to an exponential‑family distribution with a number of parameters equal to the number of constraints. In the micro‑canonical formulation the constraints must be satisfied exactly, which yields a uniform distribution over all configurations that realize a given sufficient statistic c; the probability of any admissible configuration is 1/Ω(c), where Ω(c) is the number of configurations with that statistic.

The MDL framework is employed via the Normalized Maximum Likelihood (NML) universal distribution. For any model M, the description length of data x is DL_NML = −ln L_M(x) + COMP_M, where L_M(x) is the maximum likelihood and COMP_M = ln ∑{y∈X} L_M(y) is the parametric complexity term. When a sufficient statistic exists, COMP can be expressed as a sum over the graphical values of the statistic: COMP = ln ∑{c∈C} Ω(c) L(c). This formulation makes the comparison between canonical and micro‑canonical models tractable.

The authors derive several key results:

1. Likelihood vs. Complexity Trade‑off – Micro‑canonical models always achieve a higher log‑likelihood because −ln Ω(c) is smaller than the canonical log‑likelihood for the same data. However, the complexity term for micro‑canonical models grows much faster, reflecting the combinatorial explosion of admissible configurations as the number of constraints increases.

2. Model Selection Criterion – The optimal model depends on how the observed constraint vector c* compares to the canonical model’s average fit. If the canonical model’s log‑likelihood for the observed data exceeds its uniform‑average log‑likelihood across all realizations, the canonical model yields a shorter total description length despite its lower likelihood.

3. Thermodynamic Limit and Ensemble (Non‑)Equivalence – In the limit of infinite system size, if the two ensembles are equivalent (i.e., the Kullback‑Leibler divergence D_KL between their distributions vanishes), both the likelihood difference and the complexity difference disappear, so the description‑length gap vanishes. When the number of constraints scales extensively with system size (e.g., row‑sum constraints in a bipartite network), ensemble non‑equivalence persists: D_KL remains finite, and the description‑length difference stays extensive (order N). Thus non‑equivalence manifests not only in the likelihood term but also in the complexity term.

4. Relation to Bayesian Model Selection – The paper compares the NML‑based MDL approach with Bayesian inference. In ensembles that are equivalent, the choice of prior has negligible impact on the posterior model evidence. In contrast, when many constraints are imposed and ensembles are non‑equivalent, the prior becomes crucial; different priors can lead to dramatically different description lengths and consequently different model choices. This highlights that, under non‑equivalence, priors act as implicit regularizers of model complexity.

The study provides explicit calculations for two concrete cases: (a) a single global constraint (the total number of ones in the matrix), which yields ensemble equivalence, and (b) a set of one‑sided local constraints (row sums), which generates persistent non‑equivalence. In the first case the description‑length difference vanishes asymptotically, while in the second case it remains extensive.

Overall, the paper demonstrates that canonical and micro‑canonical maximum‑entropy models should be regarded as distinct statistical models when ensemble non‑equivalence is present. Model selection must therefore incorporate both the likelihood and the complexity terms, as captured by the MDL principle. Moreover, in non‑equivalent settings, Bayesian model comparison requires careful prior specification because the prior can dominate the evidence. These insights have practical implications for any field that employs maximum‑entropy null models, suggesting that the choice between soft and hard constraints should be guided by a rigorous information‑theoretic criterion rather than convenience alone.