Active Learning for Decision Trees with Provable Guarantees

This paper advances the theoretical understanding of active learning label complexity for decision trees as binary classifiers. We make two main contributions. First, we provide the first analysis of the disagreement coefficient for decision trees-a key parameter governing active learning label complexity. Our analysis holds under two natural assumptions required for achieving polylogarithmic label complexity, (i) each root-to-leaf path queries distinct feature dimensions, and (ii) the input data has a regular, grid-like structure. We show these assumptions are essential, as relaxing them leads to polynomial label complexity. Second, we present the first general active learning algorithm for binary classification that achieves a multiplicative error guarantee, producing a $(1+ε)$-approximate classifier. By combining these results, we design an active learning algorithm for decision trees that uses only a polylogarithmic number of label queries in the dataset size, under the stated assumptions. Finally, we establish a label complexity lower bound, showing our algorithm’s dependence on the error tolerance $ε$ is close to optimal.

💡 Research Summary

This paper makes a substantial theoretical contribution to the study of active learning for decision‑tree classifiers. The authors focus on two central problems: (1) quantifying the disagreement coefficient—a key complexity measure that governs label‑complexity in many active‑learning algorithms—for the class of decision trees, and (2) designing an active‑learning algorithm that provides a multiplicative error guarantee, i.e., it returns a classifier whose error is at most (1 + ε) times the optimal error within the hypothesis class.

Disagreement coefficient analysis.
The paper first defines the hypothesis class H as the set of axis‑parallel decision trees of bounded depth d on a discrete domain X = {(a₁,…,a_dim) | a_i ∈ ℕ, a_i ≤ w}. Two structural assumptions are imposed: (i) every node on a root‑to‑leaf path tests a feature dimension that has not been used by any ancestor (distinct‑dimension paths), and (ii) the input distribution is “grid‑like,” meaning the points lie on a regular integer lattice. Under these conditions the authors prove Theorem 1.1, showing that the disagreement coefficient θ satisfies

 θ = O(log_d n)

and also provide a matching lower bound that grows polynomially when either assumption is violated. The proof proceeds by decomposing a tree into “line‑trees” (single‑leaf classifiers) and bounding the disagreement region of hypothesis balls via combinatorial arguments on the grid. This is the first explicit quantitative bound for decision trees; prior work (Balcan et al., 2010) only asserted finiteness without giving a usable estimate.

Multiplicative‑error active learning algorithm.
The second major contribution is Algorithm 2, a general active‑learning procedure for any binary classification problem that seeks a (1 + ε)‑approximate classifier with confidence 1 − δ. The algorithm maintains a version space V ⊆ H and repeatedly queries labels of points that lie in the disagreement region DIS(V). Crucially, when the version space fails to shrink quickly, the algorithm exploits the size of DIS(V) to infer a lower bound on the optimal error η, and then uses this bound to stop querying, guaranteeing the multiplicative error. Theorem 1.2 gives the label‑complexity bound

 O( ln n·θ²·(V_H·ln θ + ln ln n/δ) + θ²/ε²·(V_H·ln(θ/ε) + ln 1/δ) )

where V_H is the VC‑dimension of the hypothesis class. This bound is novel because it works in the multiplicative regime, which is stronger than the additive guarantees common in prior active‑learning literature. The authors also argue (Appendix E) that naïvely converting an additive‑error algorithm to a multiplicative one would incur a label cost Ω(n), demonstrating the necessity of a new approach.

Combining the two results for decision trees.
By plugging the θ = O(log_d n) bound from Theorem 1.1 into the general label‑complexity expression, the authors obtain a polylogarithmic query complexity for learning decision trees under the stated assumptions. Corollary 1.3 states that the number of label queries needed is

 O( ln^{2d+2} n·2^{d}(d + ln dim)·d + ln^{2d} n·(1/ε²)·2^{d}(d + dim)·ln ln n·ε + ln (1/δ) )

which is polylogarithmic in the dataset size n and only polynomial in the depth d and dimension dim. This is a dramatic improvement over naïve passive learning, which would require Θ(n) labeled examples.

Lower bound and optimality.
The paper also proves a lower bound (Theorem 4.3) showing that any active‑learning algorithm for decision trees must incur at least Ω(1/ε²) dependence on ε (up to logarithmic factors). Hence the ε‑dependence of Algorithm 2 is essentially optimal.

Context and related work.
The authors situate their work among three strands of literature: (i) realizable and agnostic active learning for classification, where most prior algorithms assume additive error; (ii) active learning for regression, where multiplicative guarantees have been explored; and (iii) theoretical analyses of decision‑tree learning, which have focused on time or sample complexity but not on active‑learning label complexity. They also discuss stronger query models (pairwise or same‑leaf queries) that achieve logarithmic label complexity under realizability, emphasizing that their model uses only standard label queries and does not assume a perfect classifier.

Strengths and limitations.
Strengths:

• First explicit quantitative bound on the disagreement coefficient for decision trees.
• Introduction of a multiplicative‑error active‑learning framework, filling a gap in the literature.
• Combination of the two results yields a polylogarithmic label‑complexity guarantee, which is near‑optimal with respect to ε.
• Clear lower‑bound arguments that justify the necessity of the structural assumptions.

Limitations:

• The distinct‑dimension path assumption and the grid‑like data distribution are restrictive; many practical datasets have correlated features and continuous domains.
• No empirical evaluation is provided, so the practical impact of the algorithm (e.g., constants hidden in the O‑notation) remains unknown.
• The analysis is confined to discrete, bounded integer domains; extending to real‑valued features would require additional technical work.

Future directions.
Potential extensions include relaxing the distinct‑dimension requirement (e.g., allowing repeated features but bounding their overlap), handling continuous or non‑grid distributions, incorporating stronger query primitives while preserving the multiplicative guarantee, and conducting empirical studies to assess real‑world performance.

In summary, the paper delivers a rigorous theoretical framework that connects the disagreement coefficient of decision trees with a novel multiplicative‑error active‑learning algorithm, achieving polylogarithmic label complexity under natural structural assumptions and establishing near‑optimal dependence on the error tolerance ε. This advances the understanding of how to efficiently label data for decision‑tree models and opens several avenues for further research.