Privately Learning Decision Lists and a Differentially Private Winnow

We give new differentially private algorithms for the classic problems of learning decision lists and large-margin halfspaces in the PAC and online models. In the PAC model, we give a computationally efficient algorithm for learning decision lists with minimal sample overhead over the best non-private algorithms. In the online model, we give a private analog of the influential Winnow algorithm for learning halfspaces with mistake bound polylogarithmic in the dimension and inverse polynomial in the margin. As an application, we describe how to privately learn decision lists in the online model, qualitatively matching state-of-the art non-private guarantees.

💡 Research Summary

This paper introduces two novel differentially private (DP) learning algorithms that achieve essentially the same statistical and computational guarantees as their non‑private counterparts for two fundamental concept classes: decision lists and large‑margin halfspaces. The contributions are presented for both the PAC (statistical) model and the online mistake‑bound model.

Private PAC learning of decision lists.
The authors revisit Rivest’s classic polynomial‑time PAC learner for decision lists, which builds the list iteratively by greedily selecting a literal that correctly classifies the remaining examples. To obtain privacy, they replace each deterministic greedy choice with a selection via the Exponential Mechanism, which samples a literal with probability proportional to exp(−ε·error/2). This yields an iterative “DP‑GreedyCover” algorithm that mirrors Rivest’s structure but respects (ε,δ)‑DP. A naïve composition analysis would suggest a sample complexity of roughly O(|F|^{3/2}), where |F| is the number of candidate features. However, the authors observe that the algorithm’s iterative structure matches that of known approximate set‑cover DP analyses, allowing a much sharper privacy accounting. Consequently, the learner uses \tilde{O}(|F|·log(1/δ)/ε) samples and runs in poly(|F|) time, matching the non‑private sample overhead up to the standard privacy factor. The result holds for generalized decision lists where each condition may be any feature from a class F, making it directly applicable to high‑dimensional, real‑world feature spaces.

Private online learning of large‑margin halfspaces (Private Winnow).
In the online setting, the classic Winnow algorithm maintains a weight vector w and updates it multiplicatively whenever a mistake occurs, achieving a mistake bound of O(log d / ρ²) for learning any ρ‑margin halfspace over {−1,1}ⁿ. To privatize this process, the authors develop a “Private Winnow” algorithm with two key innovations:

1. ConfidentWinnow – a non‑private variant that only retains the current weight vector when the prediction is both correct and “confident” (i.e., y·⟨w,x⟩ > c·ρ for a constant c). Confident predictions are robust to random sampling of w, which is crucial because the private algorithm will only have access to noisy samples of w.

2. Sparse Vector Technique – instead of updating after every mistake (which would leak the timing of errors), the algorithm aggregates errors and triggers a private update only after a privately counted threshold is exceeded. This limits the number of observable update events, preserving privacy of the error pattern.

The private algorithm samples a small number of weight vectors from the multiplicative‑weights distribution; this sampling itself is an instance of the Exponential Mechanism and therefore ε‑DP. By interleaving these samples with the Sparse Vector mechanism, the authors can treat the whole procedure as a sequential composition of standard DP primitives, enabling a clean privacy proof.

The resulting online learner is (ε,δ)‑DP against an oblivious adversary and incurs regret (or mistake) O(polylog(d,T,1/δ) / ρ⁶ ε⁴). The dependence on the margin is polynomial (ρ⁻⁶) rather than the quadratic dependence of the non‑private Winnow, but the algorithm still enjoys a polylogarithmic dependence on the ambient dimension d and on the time horizon T, matching the qualitative behavior of Winnow. This bound directly yields efficient private learning of sparse disjunctions, short decision lists, and other subclasses where Winnow is known to be attribute‑efficient.

Theoretical context and related work.
The paper situates its results within the framework of Littlestone dimension, which characterizes online learnability in the realizable case. Prior work (Golowich & Livni, 2021) showed that private online learnability is governed by this dimension; the authors leverage the fact that large‑margin halfspaces have Littlestone dimension O(log d) to obtain their polylogarithmic regret. They also discuss lower bounds showing that a Ω(log T) mistake term is unavoidable for private learning of point functions, and argue that their dependence on log T is essentially optimal.

The authors compare their methods to alternative DP approaches such as private convex optimization, private expert advice algorithms, and private query‑release techniques. Those alternatives typically incur polynomial dependence on d or T, whereas the presented algorithms achieve the desired polylogarithmic scaling. They also note that while dimension‑independent DP halfspace learners exist in the PAC setting (via dimensionality reduction or boosting), extending such results to the online model is non‑trivial and remains an open direction.

Implications and future directions.
The work demonstrates that adding differential privacy does not fundamentally increase the sample or mistake complexity for two cornerstone learning problems, provided one carefully adapts the algorithmic structure. This bridges a gap between theoretical privacy guarantees and practical, interpretable models (decision lists) used in high‑stakes domains. Future research avenues include extending the private Winnow framework to adaptive adversaries, handling agnostic (non‑realizable) settings, and empirically evaluating the algorithms on real datasets where interpretability and privacy are simultaneously required.