Scale-sensitive Psi-dimensions: the Capacity Measures for Classifiers Taking Values in R^Q

Bounds on the risk play a crucial role in statistical learning theory. They usually involve as capacity measure of the model studied the VC dimension or one of its extensions. In classification, such “VC dimensions” exist for models taking values in {0, 1}, {1,…, Q} and R. We introduce the generalizations appropriate for the missing case, the one of models with values in R^Q. This provides us with a new guaranteed risk for M-SVMs which appears superior to the existing one.

💡 Research Summary

The paper addresses a fundamental gap in statistical learning theory concerning capacity measures for classifiers whose outputs lie in the Euclidean space ℝ^Q. While the classic VC dimension and its extensions (Natarajan dimension for multiclass, Graph dimension for real‑valued functions) adequately cover models outputting binary labels, integer class indices, or scalar real numbers, they do not directly apply to vector‑valued predictors such as those used in multi‑class Support Vector Machines (M‑SVMs) or modern deep networks that produce a Q‑dimensional score vector.

To fill this gap, the authors introduce the Scale‑Sensitive Ψ‑Dimension, a unified generalization that simultaneously captures the combinatorial richness of label assignments and the magnitude‑sensitive nature of real‑valued outputs. Formally, for a hypothesis class 𝔽 ⊂ (ℝ^Q)^𝔛, a scale vector ψ = (ψ₁,…,ψ_Q) with positive components defines a ψ‑shattering condition: a set S ⊂ 𝔛 is ψ‑shattered by 𝔽 if, for every binary pattern on S, there exists a function f ∈ 𝔽 whose component‑wise deviations exceed the corresponding ψ_k thresholds, thereby realizing the pattern. The Ψ‑dimension d_Ψ(𝔽) is the largest cardinality of a ψ‑shattered set over all admissible ψ. When ψ is restricted to a single scalar, d_Ψ collapses to the classic VC dimension; when ψ takes values in {0,1}, it coincides with the Natarajan dimension; and when ψ spans a bounded interval, it reproduces the Graph dimension. Hence Ψ‑dimension subsumes all previously known combinatorial capacities.

The core theoretical contribution is a series of risk bounds that link d_Ψ to the Rademacher complexity of 𝔽. Theorem 1 shows that for any class with Ψ‑dimension d, the empirical Rademacher complexity satisfies
\