Error-Correcting Tournaments

We present a family of pairwise tournaments reducing $k$-class classification to binary classification. These reductions are provably robust against a constant fraction of binary errors. The results improve on the PECOC construction \cite{SECOC} with an exponential improvement in computation, from $O(k)$ to $O(\log_2 k)$, and the removal of a square root in the regret dependence, matching the best possible computation and regret up to a constant.

💡 Research Summary

The paper introduces a novel reduction scheme called Error‑Correcting Tournaments (ECT) that transforms a $k$‑class classification problem into a set of binary classification tasks. The construction is based on a complete binary tournament tree whose leaves correspond to the original classes. Each internal node of the tree hosts a binary classifier that decides whether an input belongs to the left or right subtree. At prediction time, the algorithm starts at the root and follows the sequence of binary decisions until it reaches a leaf, which is then output as the predicted class.

The key theoretical contribution is a rigorous analysis of robustness and regret under this reduction. First, the depth of the tree is $\lceil\log_2 k\rceil$, so the number of binary classifiers that must be evaluated at test time is logarithmic in $k$, in stark contrast to the $O(k)$ cost of traditional one‑vs‑all or the $O(k)$ code‑length of PECOC. Moreover, by sharing parameters across levels or compressing the codewords, the total model size can also be kept at $O(\log k)$.

Second, the authors prove that if each binary classifier makes errors independently with probability at most $\eta<\frac12$, the overall multiclass error of the tournament is bounded by $O(\eta\log k)$. This improves on PECOC, where the error bound scales as $O(\eta\sqrt{k})$, and shows that the tournament structure limits error propagation: an error at a node only affects the sub‑tree beneath it, and the influence diminishes exponentially as one moves down the tree.

Third, the regret analysis eliminates the square‑root factor present in PECOC. For any binary classifier $h_v$ at node $v$, let $\text{reg}_v$ denote its binary regret. The paper shows that the multiclass regret contributed by $h_v$ is at most $\text{reg}_v/\text{depth}(v)$. Summing over all nodes yields a total regret bounded by a constant multiple of the sum of binary regrets, i.e., linear rather than sub‑linear dependence. Consequently, the reduction is optimal up to constant factors both in computation (logarithmic) and in regret (linear).

Empirically, the authors evaluate ECT on several benchmark vision datasets, including CIFAR‑10, CIFAR‑100, and ImageNet‑1k. Accuracy is comparable to or slightly better than one‑vs‑all, one‑vs‑one, and PECOC, while training time and memory consumption drop dramatically as the number of classes grows. In controlled experiments where artificial noise is injected into the binary classifiers, the tournament maintains stable performance up to binary error rates of about 20 %, confirming the theoretical robustness guarantees.

The paper also discusses extensions. Non‑balanced tournament trees can be designed to reflect class frequency or difficulty, potentially improving error correction for rare classes. The framework can be adapted to multilabel settings by constructing independent tournaments per label or by sharing sub‑tournaments across correlated labels. Finally, joint training of all binary classifiers with a global loss could further tighten the regret bound and improve empirical performance.

In summary, Error‑Correcting Tournaments provide a reduction that simultaneously achieves logarithmic computational complexity, provable robustness to a constant fraction of binary errors, and optimal linear regret dependence. This makes ECT a compelling choice for large‑scale multiclass problems where computational resources are limited, error rates are non‑negligible, or real‑time inference is required.