Information, Divergence and Risk for Binary Experiments

We unify f-divergences, Bregman divergences, surrogate loss bounds (regret bounds), proper scoring rules, matching losses, cost curves, ROC-curves and information. We do this by systematically studying integral and variational representations of these objects and in so doing identify their primitives which all are related to cost-sensitive binary classification. As well as clarifying relationships between generative and discriminative views of learning, the new machinery leads to tight and more general surrogate loss bounds and generalised Pinsker inequalities relating f-divergences to variational divergence. The new viewpoint illuminates existing algorithms: it provides a new derivation of Support Vector Machines in terms of divergences and relates Maximum Mean Discrepancy to Fisher Linear Discriminants. It also suggests new techniques for estimating f-divergences.

💡 Research Summary

The paper presents a unifying framework that brings together a wide array of concepts—f‑divergences, Bregman divergences, surrogate loss (regret) bounds, proper scoring rules, cost‑sensitive loss curves, ROC curves, and information measures—under the umbrella of binary classification. Starting from a binary experiment modeled by two probability distributions (P) and (Q), the authors identify a single primitive: the cost‑sensitive 0‑1 loss (\ell_\alpha(t)=\alpha\mathbf{1}{t<0}+(1-\alpha)\mathbf{1}{t\ge0}). By applying integral and variational transformations to (\ell_\alpha), they show that every f‑divergence (D_f(P|Q)) and every Bregman divergence (B_\phi(P|Q)) can be expressed as an integral kernel built from (\ell_\alpha). In particular, the convex function (f) is obtained as a Laplace‑type transform of (\ell_\alpha), while (\phi) emerges from the Fenchel conjugate of the same loss.

The variational representation plays a central role:
\