PENEX: AdaBoost-Inspired Neural Network Regularization

AdaBoost sequentially fits so-called weak learners to minimize an exponential loss, which penalizes misclassified data points more severely than other loss functions like cross-entropy. Paradoxically, AdaBoost generalizes well in practice as the number of weak learners grows. In the present work, we introduce Penalized Exponential Loss (PENEX), a new formulation of the multi-class exponential loss that is theoretically grounded and, in contrast to the existing formulation, amenable to optimization via first-order methods, making it a practical objective for training neural networks. We demonstrate that PENEX effectively increases margins of data points, which can be translated into a generalization bound. Empirically, across computer vision and language tasks, PENEX improves neural network generalization in low-data regimes, often matching or outperforming established regularizers at comparable computational cost. Our results highlight the potential of the exponential loss beyond its application in AdaBoost.

💡 Research Summary

The paper introduces PENEX (Penalized Exponential Loss), a novel multi‑class exponential loss designed to bring the regularizing benefits of AdaBoost’s exponential loss into modern deep neural network training. AdaBoost famously minimizes an exponential loss by sequentially adding weak learners (e.g., decision stumps). Despite the aggressive penalization of mis‑classifications, AdaBoost often improves generalization as more weak learners are added—a phenomenon traditionally explained by margin theory. However, directly applying the classic multi‑class exponential loss (CONEX) to neural networks is problematic because it imposes a hard zero‑sum constraint on logits, requiring sophisticated constrained optimization techniques.

PENEX replaces this hard constraint with a smooth SumExp penalty. Formally, for logits f(x)∈ℝ^K, the loss is

L_PENEX(f;α,ρ)=E