Laws of Learning Dynamics and the Core of Learners

We formulate the fundamental laws governing learning dynamics, namely the conservation law and the decrease of total entropy. Within this framework, we introduce an entropy-based lifelong ensemble learning method. We evaluate its effectiveness by constructing an immunization mechanism to defend against transfer-based adversarial attacks on the CIFAR-10 dataset. Compared with a naive ensemble formed by simply averaging models specialized on clean and adversarial samples, the resulting logifold achieves higher accuracy in most test cases, with particularly large gains under strong perturbations.

💡 Research Summary

The paper “Laws of Learning Dynamics and the Core of Learners” proposes a theoretical framework that treats the learning process of machine‑learning models in a manner analogous to the laws of thermodynamics. The authors first formalize a single model as a pair ((\Phi, f)) where (\Phi) maps a subset of the input space into a Euclidean feature space and (f) maps that feature space to a probability simplex over a finite label set. They define the entropy of a model (H(\Phi,f)) as the integral of Shannon entropy of the output distribution over the domain of (\Phi) plus a maximal‑entropy term for the uncovered part of the input space. The cross‑entropy (H(T,(\Phi,f))) measures the discrepancy between the model and the true labeling function (T).

Two fundamental “laws” are derived:

1. Conservation Law of Learning (Proposition 2.4). The work done by the gradient of the model entropy against the difference between truth and model equals the difference between cross‑entropy and entropy. Symbolically, \