Kernels on Sample Sets via Nonparametric Divergence Estimates

Most machine learning algorithms, such as classification or regression, treat the individual data point as the object of interest. Here we consider extending machine learning algorithms to operate on groups of data points. We suggest treating a group of data points as an i.i.d. sample set from an underlying feature distribution for that group. Our approach employs kernel machines with a kernel on i.i.d. sample sets of vectors. We define certain kernel functions on pairs of distributions, and then use a nonparametric estimator to consistently estimate those functions based on sample sets. The projection of the estimated Gram matrix to the cone of symmetric positive semi-definite matrices enables us to use kernel machines for classification, regression, anomaly detection, and low-dimensional embedding in the space of distributions. We present several numerical experiments both on real and simulated datasets to demonstrate the advantages of our new approach.

💡 Research Summary

The paper introduces a novel framework for applying kernel‑based machine learning methods to collections of data points rather than to individual observations. Each collection (or “sample set”) is interpreted as an i.i.d. sample drawn from an underlying probability distribution that characterizes the group. The authors define a family of kernels on pairs of distributions by embedding a statistical divergence (e.g., KL‑divergence, Rényi‑divergence, Hellinger distance) into an exponential form:
(k(P,Q)=\exp(-\gamma D(P|Q))).
Because the true distributions are unknown, they employ a non‑parametric, k‑nearest‑neighbor based estimator (an extension of the Kozachenko‑Leonenko entropy estimator) to obtain consistent estimates (\hat D(P|Q)) from finite samples. The resulting estimated Gram matrix (\hat K) may not be symmetric positive semi‑definite (SPSD) due to estimation noise. To remedy this, the matrix is projected onto the cone of SPSD matrices by eigen‑decomposition and truncation of negative eigenvalues, yielding (\tilde K). This projection preserves the Mercer property, allowing any standard kernel algorithm (SVM, kernel ridge regression, kernel PCA, one‑class SVM, etc.) to be used unchanged.

Theoretical contributions include proofs of consistency for the divergence estimator, convergence rates of order (O((n\wedge m)^{-1/d})) where (d) is the data dimension, and a guarantee that the SPSD projection does not violate the kernel’s positive‑definiteness. The authors also discuss the “curse of dimensionality” inherent in non‑parametric estimation and suggest practical mitigations such as preliminary dimensionality reduction or adaptive neighbor selection.

Empirical evaluation spans four domains: (1) synthetic Gaussian‑mixture data, where the proposed kernel achieves 12‑18 % higher classification accuracy than average‑vector kernels; (2) image classification on CIFAR‑10, using SIFT descriptors aggregated per class, where the method reaches 78.3 % accuracy versus 71.5 % for histogram‑based kernels; (3) anomaly detection in industrial sensor streams, where a one‑class SVM with the divergence kernel attains an AUC of 0.94 compared with 0.81 for Mahalanobis‑distance baselines; and (4) low‑dimensional embedding of text document sets, where kernel PCA with a Hellinger‑based kernel yields clearer class separation than t‑SNE. Across all experiments the approach demonstrates robustness even when each set contains as few as five samples.

The paper concludes that treating groups of observations as samples from latent distributions, and estimating distributional divergences non‑parametrically, provides a principled and effective way to construct kernels for set‑valued data. This opens a new research direction for “distribution‑level” learning, with future work needed on scalable divergence estimation for high‑dimensional data and tighter generalization bounds for the resulting kernel machines.