Asymptotic Learning Curve and Renormalizable Condition in Statistical Learning Theory

Bayes statistics and statistical physics have the common mathematical structure, where the log likelihood function corresponds to the random Hamiltonian. Recently, it was discovered that the asymptotic learning curves in Bayes estimation are subject to a universal law, even if the log likelihood function can not be approximated by any quadratic form. However, it is left unknown what mathematical property ensures such a universal law. In this paper, we define a renormalizable condition of the statistical estimation problem, and show that, under such a condition, the asymptotic learning curves are ensured to be subject to the universal law, even if the true distribution is unrealizable and singular for a statistical model. Also we study a nonrenormalizable case, in which the learning curves have the different asymptotic behaviors from the universal law.

💡 Research Summary

The paper investigates why Bayes‑based learning curves obey a universal asymptotic law even when the underlying statistical model is singular or unrealizable, i.e., when the log‑likelihood cannot be approximated by a quadratic form. By drawing an analogy between Bayes statistics and statistical physics—where the log‑likelihood plays the role of a random Hamiltonian—the authors introduce a “renormalizable condition” that captures the essential mathematical property guaranteeing the universal law.

The renormalizable condition consists of two parts. First, in a sufficiently small neighbourhood of the true parameter, the expected loss K(w) must be bounded above by a constant multiple of the log‑likelihood L(w) (K(w) ≤ C·L(w) for some C>0). This ensures that the loss does not become arbitrarily smaller than the log‑likelihood, mirroring the renormalization concept in physics where high‑energy divergences are controlled. Second, the loss function must admit well‑defined algebraic‑geometric invariants: the real log canonical threshold λ and the real multiplicity ν. These invariants measure the dimensionality and complexity of the singularities of the model.

When the condition holds, the authors prove that the Bayes generalization error Gₙ satisfies
 E