PAC learnability of a concept class under non-atomic measures: a problem by Vidyasagar

In response to a 1997 problem of M. Vidyasagar, we state a necessary and sufficient condition for distribution-free PAC learnability of a concept class $\mathscr C$ under the family of all non-atomic (diffuse) measures on the domain $\Omega$. Clearly, finiteness of the classical Vapnik-Chervonenkis dimension of $\mathscr C$ is a sufficient, but no longer necessary, condition. Besides, learnability of $\mathscr C$ under non-atomic measures does not imply the uniform Glivenko-Cantelli property with regard to non-atomic measures. Our learnability criterion is stated in terms of a combinatorial parameter $\VC({\mathscr C},{\mathrm{mod}},\omega_1)$ which we call the VC dimension of $\mathscr C$ modulo countable sets. The new parameter is obtained by thickening up'' single points in the definition of VC dimension to uncountable clusters’’. Equivalently, $\VC(\mathscr C\modd\omega_1)\leq d$ if and only if every countable subclass of $\mathscr C$ has VC dimension $\leq d$ outside a countable subset of $\Omega$. The new parameter can be also expressed as the classical VC dimension of $\mathscr C$ calculated on a suitable subset of a compactification of $\Omega$. We do not make any measurability assumptions on $\mathscr C$, assuming instead the validity of Martin’s Axiom (MA).

💡 Research Summary

The paper addresses a problem posed by M. Vidyasagar in 1997: to give a combinatorial characterization of concept classes that are distribution‑free PAC learnable when the underlying probability measures are restricted to the family of all non‑atomic (diffuse) measures on a domain Ω. In the classical setting, three conditions are equivalent for a concept class C: (1) distribution‑free PAC learnability (i.e., learnability with respect to the set P(Ω) of all probability measures), (2) the uniform Glivenko‑Cantelli property with respect to P(Ω), and (3) finiteness of the Vapnik–Chervonenkis (VC) dimension. The authors show that when the family of measures is narrowed to the non‑atomic ones, condition (3) remains sufficient but is no longer necessary.

To capture the exact boundary of learnability under non‑atomic measures, the authors introduce a new combinatorial parameter, denoted VC(C mod ω₁), which they call the VC dimension of C modulo countable sets. Intuitively, one “thickens” each point in the usual VC definition into an uncountable cluster, thereby ignoring countable perturbations that are invisible to non‑atomic measures. Formally, VC(C mod ω₁) ≤ d if and only if for every countable subclass C′ ⊆ C there exists a countable set N(C′) ⊂ Ω such that the restriction of C′ to Ω \ N(C′) has ordinary VC dimension at most d. Equivalently, VC(C mod ω₁) is the ordinary VC dimension of C when the domain is replaced by a suitable subspace of the Stone‑Čech compactification βΩ obtained by quotienting out the ideal of countable sets.

The main theorem (Theorem 1) establishes the equivalence of seven statements for a concept class C on a standard Borel space, assuming Martin’s Axiom (MA). The statements are:

1. C is PAC learnable with respect to the family P_na(Ω) of all non‑atomic probability measures.
2. VC(C mod ω₁) = d < ∞.
3. Every countable subclass C′ ⊆ C has finite VC dimension on the complement of some countable subset of Ω (the countable set may depend on C′).
4. There exists a uniform bound d such that the condition in (3) holds for all countable C′.
5. Every countable subclass C′ is a uniform Glivenko‑Cantelli class for non‑atomic measures.
6. The sample complexity s(ε,δ) of a learning rule depends only on C (not on the particular countable subclass).
7. If C is universally separable, then (2)–(6) are also equivalent to: (7) VC dimension of C is finite outside a countable subset of Ω, and (8) C is a uniform Glivenko‑Cantelli class for non‑atomic measures.

The implication (3) ⇒ (1) is the technical heart of the paper. The authors construct a consistent learning rule L whose image on any sample (σ, C∩σ) forms a uniform Glivenko‑Cantelli class. This construction relies on MA to guarantee the existence of enough ultrafilters (free ultrafilters on Ω) that avoid the countable ideal, thereby allowing one to “ignore” countable noise while still retaining enough combinatorial richness to guarantee uniform convergence.

The paper also provides a Boolean‑algebraic perspective. By viewing Ω’s power set as a Boolean algebra B = 2^Ω and letting I be the ideal of countable subsets, the quotient algebra B/I yields a Stone space S(2^Ω/I). The VC dimension of C modulo I, VC(C mod I), is defined as the ordinary VC dimension of the family of closed sets {cl(C) : C ∈ C} restricted to S(2^Ω/I). Theorem 2 shows that VC(C mod I) ≥ n precisely when there exist n measurable subsets A₁,…,A_n not in I that are shattered by C in the sense that for each J ⊆ {1,…,n} there is a concept C_J ∈ C containing all A_i with i∈J and disjoint from all A_i with i∉J. This equivalence connects the new parameter to classical VC theory and demonstrates that compactifying Ω (e.g., passing to βΩ) does not change the VC dimension.

An illustrative counterexample is given: let C consist of all finite and co‑finite subsets of a standard Borel space Ω. Then VC(C) = ∞, yet VC(C mod ω₁) = 1, and C is PAC learnable under non‑atomic measures (a trivial learner that distinguishes only ∅ and Ω suffices). This shows that PAC learnability under non‑atomic measures does not imply the uniform Glivenko‑Cantelli property, and that the new parameter captures precisely the learnability boundary.

In summary, the paper extends the classical PAC learning theory to the setting of non‑atomic measures by introducing the VC dimension modulo countable sets, proving its equivalence to distribution‑free PAC learnability under MA, and providing both combinatorial and Boolean‑algebraic characterizations. The results clarify that finiteness of the ordinary VC dimension is only a sufficient condition in this restricted measure class, and they open the way for further investigations of learning under intermediate families of measures.