Statistical Learning Theory in Lean 4: Empirical Processes from Scratch

We present the first comprehensive Lean 4 formalization of statistical learning theory (SLT) grounded in empirical process theory. Our end-to-end formal infrastructure implement the missing contents in latest Lean 4 Mathlib library, including a complete development of Gaussian Lipschitz concentration, the first formalization of Dudley’s entropy integral theorem for sub-Gaussian processes, and an application to least-squares (sparse) regression with a sharp rate. The project was carried out using a human-AI collaborative workflow, in which humans design proof strategies and AI agents execute tactical proof construction, leading to the human-verified Lean 4 toolbox for SLT. Beyond implementation, the formalization process exposes and resolves implicit assumptions and missing details in standard SLT textbooks, enforcing a granular, line-by-line understanding of the theory. This work establishes a reusable formal foundation and opens the door for future developments in machine learning theory. The code is available at https://github.com/YuanheZ/lean-stat-learning-theory

💡 Research Summary

This paper presents the first comprehensive formalization of statistical learning theory (SLT) in the interactive theorem prover Lean 4, building a complete end‑to‑end infrastructure that was previously missing from the Lean 4 Mathlib library. The authors focus on three technical pillars: (1) Gaussian Lipschitz concentration, (2) Dudley’s entropy integral for sub‑Gaussian processes, and (3) an application to least‑squares (including ℓ₁‑constrained) regression that yields sharp, minimax‑optimal rates.

To obtain Gaussian Lipschitz concentration, the work develops a cascade of high‑dimensional Gaussian tools: a fully formal Efron‑Stein inequality that works with heterogeneous coordinate distributions, the Gaussian Poincaré inequality, a density‑approximation argument, and finally the Gaussian logarithmic Sobolev inequality (LSI). Crucially, the authors construct the necessary measure‑theoretic machinery (product measures, conditional expectations, measure rectangles) and Sobolev spaces W¹,²(γ⊗n), proving that smooth compactly supported functions are dense in these spaces. This chain of results, never before formalized in any theorem prover, enables the extension of concentration results from C∞c functions to the full class of Lipschitz functions.

The second pillar, Dudley’s entropy integral, is formalized for general sub‑Gaussian processes. The authors define covering and packing numbers in arbitrary metric spaces, develop dyadic approximations, and implement the chaining argument that decomposes a stochastic process into a telescoping sum over successive approximations. They also reconcile different notions of integration in Lean (Bochner vs. interval integrals) to make the entropy bound precise. The resulting theorem states that the expected supremum of a sub‑Gaussian process is bounded by an integral of the square root of the log‑covering numbers, exactly as in the classical Dudley bound.

With these tools in place, the paper demonstrates a unified “localized empirical process” framework for regression. By restricting attention to a δ‑neighborhood of the optimal predictor, the authors bound the localized Gaussian complexity via Dudley’s integral, solve for the critical radius δ⋆, and plug it into a master error bound. This yields sharp excess‑risk rates for ordinary linear regression and for ℓ₁‑constrained (sparse) regression, matching known minimax lower bounds. The approach improves upon prior formalizations that relied only on Rademacher complexity and produced suboptimal rates.

Beyond the mathematics, the project showcases a human‑AI collaborative workflow. Human experts design proof strategies, decompose complex theorems into lemmas, and guide the overall architecture. AI agents (Claude Code and Opus‑4.5) execute the tactical proof steps, generate Lean code, and iteratively refine the formalization under human supervision. The entire effort required roughly 500 supervised hours and produced about 30 000 lines of Lean 4 code, all compiled without any “sorry” placeholders or additional axioms. This demonstrates that large‑scale formalization of modern machine‑learning theory, traditionally thought to require years of expert effort, can be dramatically accelerated through careful human‑AI partnership.

The authors also reflect on the pedagogical value of the formalization: every definition, assumption, measurability condition, and topological nuance that textbooks often gloss over becomes explicit in Lean. Consequently, the library serves not only as a verification tool but also as a training ground for students to acquire deep, line‑by‑line understanding of SLT.

In summary, the paper delivers a reusable, machine‑checked foundation for statistical learning theory, fills critical gaps in Lean’s probability and analysis libraries, and opens the door for future formal developments in deep learning, high‑dimensional statistics, and related fields. The combination of rigorous mathematics, extensive engineering, and innovative human‑AI collaboration makes this work a landmark contribution to both the formal methods community and the theoretical machine‑learning community.