From average case complexity to improper learning complexity

The basic problem in the PAC model of computational learning theory is to determine which hypothesis classes are efficiently learnable. There is presently a dearth of results showing hardness of learning problems. Moreover, the existing lower bounds fall short of the best known algorithms. The biggest challenge in proving complexity results is to establish hardness of {\em improper learning} (a.k.a. representation independent learning).The difficulty in proving lower bounds for improper learning is that the standard reductions from $\mathbf{NP}$-hard problems do not seem to apply in this context. There is essentially only one known approach to proving lower bounds on improper learning. It was initiated in (Kearns and Valiant 89) and relies on cryptographic assumptions. We introduce a new technique for proving hardness of improper learning, based on reductions from problems that are hard on average. We put forward a (fairly strong) generalization of Feige’s assumption (Feige 02) about the complexity of refuting random constraint satisfaction problems. Combining this assumption with our new technique yields far reaching implications. In particular, 1. Learning $\mathrm{DNF}$’s is hard. 2. Agnostically learning halfspaces with a constant approximation ratio is hard. 3. Learning an intersection of $\omega(1)$ halfspaces is hard.

💡 Research Summary

The paper addresses a central open problem in computational learning theory: establishing strong hardness results for improper (representation‑independent) PAC learning. While many proper‑learning hardness results follow from NP‑hardness of distinguishing realizable from unrealizable samples, such reductions fail for improper learning because the learner may output hypotheses outside the target class. Historically, the only successful approach for improper learning hardness has relied on cryptographic assumptions (e.g., one‑way trapdoor permutations), which are strong and often unrelated to the combinatorial structure of the hypothesis class.

The authors propose a fundamentally different methodology based on average‑case hardness of constraint satisfaction problems (CSPs). The key insight is that if it is computationally difficult to distinguish a “realizable” sample (coming from a distribution that can be perfectly satisfied by some hypothesis in the class) from a random, unsatisfiable sample, then any improper learner would also fail to achieve non‑trivial performance. To formalize this, they introduce a generalized version of Feige’s conjecture on the hardness of refuting random 3‑SAT formulas. In the original conjecture, random 3‑SAT instances with a linear number of clauses are assumed to be hard to refute (i.e., to certify unsatisfiability) in polynomial time. The authors extend this to arbitrary Boolean predicates P, defining the problem CSP₁,randₘ(n)(P): given either a satisfiable instance of CSP(P) or a random instance with m(n) clauses, decide which case holds. Their generalized assumption states that for a broad family of predicates, no polynomial‑time algorithm can succeed with noticeable advantage.

Using this average‑case assumption, the paper constructs reductions from CSP₁,randₘ(n)(P) to several natural learning problems, thereby transferring the presumed hardness. The reductions are carefully designed so that a successful improper learner would yield an algorithm that distinguishes satisfiable CSP instances from random ones, contradicting the assumption.

The main results are:

1. Hardness of learning DNF formulas – By reducing random 3‑SAT (or a suitable predicate) to the problem of learning DNF under the uniform distribution on {±1}ⁿ, the authors show that any polynomial‑time improper learner for DNF would refute random 3‑SAT, violating the generalized Feige assumption. Consequently, learning DNF is hard even when the learner is allowed to output arbitrary hypotheses.

2. Hardness of agnostic learning of halfspaces with constant approximation – The paper maps instances of CSP(P) to labeled examples for halfspace learning over the Boolean cube. It proves that achieving any constant‑factor approximation (i.e., outputting a hypothesis whose error is at most α·OPT + ε for some constant α) would enable refutation of random CSP instances. Hence, constant‑approximation agnostic learning of halfspaces is computationally infeasible under the average‑case assumption, even when the input space is the Boolean cube.

3. Hardness of learning intersections of ω(1) halfspaces – Extending the previous reduction, the authors show that learning the intersection of a super‑constant number of halfspaces (the class of points satisfying all of the halfspaces) is also hard for improper learners. The reduction again leverages the inability to distinguish satisfiable CSP instances from random ones.

Additional corollaries include hardness of learning finite automata (recoverable via the cryptographic technique) and parity functions (a standard assumption in learning theory). The authors also conjecture that learning intersections of a constant number of halfspaces should be hard under the same framework, and they outline a potential approach for the case of four halfspaces.

The paper contrasts its average‑case approach with the cryptographic technique. While cryptographic reductions require a problem and a distribution that fool all algorithms (a very strong requirement), the average‑case method only needs to construct a distribution that defeats any specific learning algorithm, which is a weaker and more natural condition for learning problems. Moreover, CSPs are directly related to the structure of many hypothesis classes, making the reductions more transparent.

In summary, the work introduces a powerful new paradigm: leveraging average‑case hardness of CSPs to prove impossibility results for improper PAC learning. This bridges a gap between average‑case complexity, hardness of approximation, and learning theory, and yields the strongest known hardness results for several fundamental learning tasks, including DNF formulas, halfspaces, and intersections of many halfspaces. The methodology opens a promising avenue for future research on the complexity of learning other natural hypothesis classes.