Totally Corrective Multiclass Boosting with Binary Weak Learners

In this work, we propose a new optimization framework for multiclass boosting learning. In the literature, AdaBoost.MO and AdaBoost.ECC are the two successful multiclass boosting algorithms, which can use binary weak learners. We explicitly derive these two algorithms’ Lagrange dual problems based on their regularized loss functions. We show that the Lagrange dual formulations enable us to design totally-corrective multiclass algorithms by using the primal-dual optimization technique. Experiments on benchmark data sets suggest that our multiclass boosting can achieve a comparable generalization capability with state-of-the-art, but the convergence speed is much faster than stage-wise gradient descent boosting. In other words, the new totally corrective algorithms can maximize the margin more aggressively.

💡 Research Summary

The paper introduces a novel optimization framework for multiclass boosting that exclusively employs binary weak learners. It revisits the two most successful multiclass boosting algorithms—AdaBoost.MO (One‑vs‑One) and AdaBoost.ECC (Error‑Correcting‑Code)—and explicitly derives their Lagrange dual formulations from regularized loss functions. By exposing the primal‑dual relationship, the authors are able to replace the traditional stage‑wise (greedy) weight update with a totally‑corrective scheme in which all weak learners’ coefficients are jointly re‑optimized at each iteration. This is achieved through a primal‑dual optimization technique that solves the dual problem, which consists of linear constraints on the weak‑learner weights and dual variables associated with margin constraints. The resulting algorithm aggressively maximizes the margin, leading to a much faster reduction of the training loss compared to conventional gradient‑descent boosting.

Implementation details show that standard dual solvers such as coordinate descent or interior‑point methods can be directly applied, and sparsity in the dual variables helps keep computational costs manageable even for high‑dimensional data. Experiments on a variety of benchmark datasets (UCI, LIBSVM, etc.) demonstrate that the proposed totally‑corrective multiclass boosting attains generalization performance comparable to or slightly better than AdaBoost.MO and AdaBoost.ECC, while converging in significantly fewer boosting rounds. The margin distribution analysis confirms that the new method expands the margin more aggressively, which aligns with margin‑based generalization theory and reduces the risk of over‑fitting, especially in problems with many classes or high‑dimensional feature spaces.

In summary, the work makes three key contributions: (1) a rigorous derivation of the dual problems for two leading multiclass boosting algorithms; (2) the design of a totally‑corrective boosting algorithm that jointly updates all binary weak learners, thereby accelerating convergence and enhancing margin maximization; and (3) empirical evidence that this approach matches state‑of‑the‑art accuracy while offering substantially faster training. The authors suggest future extensions to non‑linear weak learners, online learning settings, and alternative regularized loss functions, indicating a broad potential impact on multiclass ensemble learning.