Boosting through Optimization of Margin Distributions

Boosting has attracted much research attention in the past decade. The success of boosting algorithms may be interpreted in terms of the margin theory. Recently it has been shown that generalization error of classifiers can be obtained by explicitly taking the margin distribution of the training data into account. Most of the current boosting algorithms in practice usually optimizes a convex loss function and do not make use of the margin distribution. In this work we design a new boosting algorithm, termed margin-distribution boosting (MDBoost), which directly maximizes the average margin and minimizes the margin variance simultaneously. This way the margin distribution is optimized. A totally-corrective optimization algorithm based on column generation is proposed to implement MDBoost. Experiments on UCI datasets show that MDBoost outperforms AdaBoost and LPBoost in most cases.

💡 Research Summary

The paper addresses a fundamental limitation of most existing boosting algorithms: they optimize a convex loss (e.g., exponential loss in AdaBoost or a linear program in LPBoost) without explicitly considering the full distribution of margins on the training set. Recent theoretical work has shown that the generalization error of a classifier can be tightly bounded not only by the smallest margin but also by statistics of the entire margin distribution, such as its mean and variance. Motivated by this insight, the authors propose a new boosting method called Margin‑Distribution Boosting (MDBoost) that directly maximizes the average margin while simultaneously minimizing the margin variance. By doing so, MDBoost seeks a margin distribution that is both shifted to the right (higher confidence) and concentrated (lower variability), which is expected to improve robustness and reduce over‑fitting.

The algorithm is formulated as a joint optimization problem over the weights of the weak learners (α) and the classifier output f(x)=∑jαjhj(x). The objective function is
\