Distribution-Specific Agnostic Boosting

We consider the problem of boosting the accuracy of weak learning algorithms in the agnostic learning framework of Haussler (1992) and Kearns et al. (1992). Known algorithms for this problem (Ben-David et al., 2001; Gavinsky, 2002; Kalai et al., 2008) follow the same strategy as boosting algorithms in the PAC model: the weak learner is executed on the same target function but over different distributions on the domain. We demonstrate boosting algorithms for the agnostic learning framework that only modify the distribution on the labels of the points (or, equivalently, modify the target function). This allows boosting a distribution-specific weak agnostic learner to a strong agnostic learner with respect to the same distribution. When applied to the weak agnostic parity learning algorithm of Goldreich and Levin (1989) our algorithm yields a simple PAC learning algorithm for DNF and an agnostic learning algorithm for decision trees over the uniform distribution using membership queries. These results substantially simplify Jackson’s famous DNF learning algorithm (1994) and the recent result of Gopalan et al. (2008). We also strengthen the connection to hard-core set constructions discovered by Klivans and Servedio (1999) by demonstrating that hard-core set constructions that achieve the optimal hard-core set size (given by Holenstein (2005) and Barak et al. (2009)) imply distribution-specific agnostic boosting algorithms. Conversely, our boosting algorithm gives a simple hard-core set construction with an (almost) optimal hard-core set size.

💡 Research Summary

The paper introduces a novel boosting technique for agnostic learning that departs from the traditional approach of re‑weighting the input distribution. In the classic agnostic boosting framework (e.g., Ben‑David et al., 2001; Gavinsky, 2002; Kalai et al., 2008), a weak learner is invoked repeatedly on the same target function but under a sequence of carefully crafted distributions over the instance space. This requires a “distribution‑independent” weak learner, which can be difficult to obtain in practice. The authors propose instead to keep the underlying data distribution fixed and to modify only the distribution of the labels (equivalently, to modify the target function) at each boosting round. By flipping the label of each example with a probability that depends on the current error of the hypothesis, they generate a new “noisy‑label” distribution that is more favorable for the weak learner. The key insight is that a weak agnostic learner that succeeds with advantage γ on a fixed distribution D can be amplified to a strong learner on the same D using only label‑noise transformations. The analysis shows that after O((1/γ)·log(1/ε)) rounds the error drops to ε, matching the guarantees of standard boosting while avoiding any change to the instance distribution.

A central application of this framework is the use of the Goldreich‑Levin algorithm (1989), a classic weak learner for parity functions under the uniform distribution. The authors show that, when combined with their label‑only boosting, Goldreich‑Levin yields a simple and efficient PAC learner for DNF formulas and an agnostic learner for decision trees, both over the uniform distribution and using membership queries. This dramatically simplifies Jackson’s celebrated DNF learning algorithm (1994), which relied on Fourier analysis, boosting, and a more intricate use of membership queries. The new approach reduces the whole pipeline to a single call to Goldreich‑Levin followed by a few rounds of label‑noise boosting, achieving comparable sample and time complexity with a far more transparent implementation. The same technique also reproduces the recent results of Gopalan et al. (2008) on agnostic decision‑tree learning, again with a cleaner algorithmic structure.

Beyond algorithmic contributions, the paper deepens the theoretical connection between agnostic boosting and hard‑core set constructions. Klivans and Servedio (1999) observed that boosting can be interpreted as constructing a hard‑core set, but their construction did not achieve the optimal size. Holenstein (2005) and Barak et al. (2009) later proved that the optimal hard‑core set size is Θ(1/ε). The authors demonstrate that any optimal‑size hard‑core set immediately yields a distribution‑specific agnostic boosting algorithm of the type they propose. Conversely, their boosting procedure itself gives a straightforward construction of a hard‑core set whose size is within a constant factor of the optimal bound. This bidirectional relationship unifies two previously separate strands of complexity‑theoretic research and suggests that advances in one area can be translated directly into the other.

In summary, the paper makes four major contributions: (1) a label‑only agnostic boosting framework that works with distribution‑specific weak learners; (2) a rigorous analysis showing geometric error reduction and logarithmic dependence on the target accuracy; (3) concrete applications that simplify and improve learning algorithms for DNF formulas and decision trees under the uniform distribution; and (4) a tight link between optimal hard‑core set constructions and agnostic boosting, providing both a new use for known hard‑core results and a simple method for constructing near‑optimal hard‑cores. These results broaden the toolkit for agnostic learning, lower the barrier to implementing practical boosting algorithms, and illuminate the deep structural connections between learning theory and computational hardness.