Efficiency versus Convergence of Boolean Kernels for On-Line Learning Algorithms

The paper studies machine learning problems where each example is described using a set of Boolean features and where hypotheses are represented by linear threshold elements. One method of increasing the expressiveness of learned hypotheses in this context is to expand the feature set to include conjunctions of basic features. This can be done explicitly or where possible by using a kernel function. Focusing on the well known Perceptron and Winnow algorithms, the paper demonstrates a tradeoff between the computational efficiency with which the algorithm can be run over the expanded feature space and the generalization ability of the corresponding learning algorithm. We first describe several kernel functions which capture either limited forms of conjunctions or all conjunctions. We show that these kernels can be used to efficiently run the Perceptron algorithm over a feature space of exponentially many conjunctions; however we also show that using such kernels, the Perceptron algorithm can provably make an exponential number of mistakes even when learning simple functions. We then consider the question of whether kernel functions can analogously be used to run the multiplicative-update Winnow algorithm over an expanded feature space of exponentially many conjunctions. Known upper bounds imply that the Winnow algorithm can learn Disjunctive Normal Form (DNF) formulae with a polynomial mistake bound in this setting. However, we prove that it is computationally hard to simulate Winnows behavior for learning DNF over such a feature set. This implies that the kernel functions which correspond to running Winnow for this problem are not efficiently computable, and that there is no general construction that can run Winnow with kernels.

💡 Research Summary

The paper investigates the trade‑off between computational efficiency and convergence speed when applying linear threshold models—specifically the Perceptron and Winnow algorithms—to Boolean data that have been expanded with conjunction (feature) terms. The authors first introduce kernel functions that implicitly represent all possible conjunctions (including both positive and negative literals) or restricted subsets (e.g., monotone conjunctions, conjunctions of bounded size). By using these kernels, the Perceptron can be simulated over an exponentially large feature space (up to 3ⁿ dimensions) in polynomial time per example, as formalized in Theorem 3. However, the classic Perceptron mistake bound depends on the norm of the target weight vector and the radius of the examples; when the feature space is expanded, both quantities become exponential, leading to an exponential upper bound on the number of mistakes. The authors construct a concrete monotone DNF (essentially a single full conjunction) and a carefully designed sequence of examples that forces the kernel Perceptron to make Ω(n) mistakes, even when the target function is extremely simple. This lower bound holds for variants with adaptive thresholds or learning rates, showing that kernel Perceptron, while computationally cheap, suffers from prohibitively slow convergence on rich Boolean feature spaces.

Turning to Winnow, the paper recalls its multiplicative update rule (promotion factor α > 1 and threshold θ) and the known logarithmic mistake bound for learning monotone disjunctions (Theorem 2). In principle, if a kernel could simulate Winnow over all conjunctions, the bound would imply a polynomial‑mistake algorithm for learning DNF formulas. The authors prove, however, that such a kernel cannot be computed efficiently. They reduce the existence of an efficient Winnow kernel to solving a #P‑complete counting problem (essentially counting satisfying assignments of a Boolean formula). Consequently, unless #P collapses to P, no polynomial‑time algorithm can simulate Winnow over an exponential number of conjunctions. This establishes a stark contrast: Winnow enjoys rapid convergence theoretically, but its kernel implementation is computationally intractable.

Overall, the paper demonstrates a fundamental efficiency‑versus‑convergence trade‑off for Boolean kernels. Perceptron kernels are fast to compute but converge slowly; Winnow kernels converge quickly but are computationally hard to realize. The results justify practical systems (e.g., SNoW) that restrict the conjunction size or use limited feature expansions, and they highlight the need for new kernel constructions or approximation techniques if one wishes to combine the best of both worlds for learning complex Boolean concepts such as DNF.