Some Robustness Properties of Label Cleaning

We demonstrate that learning procedures that rely on aggregated labels, e.g., label information distilled from noisy responses, enjoy robustness properties impossible without data cleaning. This robustness appears in several ways. In the context of risk consistency – when one takes the standard approach in machine learning of minimizing a surrogate (typically convex) loss in place of a desired task loss (such as the zero-one mis-classification error) – procedures using label aggregation obtain stronger consistency guarantees than those even possible using raw labels. And while classical statistical scenarios of fitting perfectly-specified models suggest that incorporating all possible information – modeling uncertainty in labels – is statistically efficient, consistency fails for ``standard’’ approaches as soon as a loss to be minimized is even slightly mis-specified. Yet procedures leveraging aggregated information still converge to optimal classifiers, highlighting how incorporating a fuller view of the data analysis pipeline, from collection to model-fitting to prediction time, can yield a more robust methodology by refining noisy signals.

💡 Research Summary

This paper investigates how aggregating noisy labels—a process often called label cleaning—affects the theoretical guarantees of supervised learning, especially the consistency of surrogate risk minimization. Classical learning theory assumes we observe independent pairs (X, Y) and minimize a surrogate loss φ that is easier to optimize than the target loss ℓ (e.g., 0‑1 loss). Consistency is usually expressed in two ways: Fisher consistency (if the surrogate risk is minimized, the target risk is also minimized) and a stronger uniform comparison inequality that links excess surrogate risk to excess target risk via a calibration function ψ.

The authors argue that modern data collection pipelines rarely provide a single clean label per example; instead, multiple noisy annotations are collected (crowdsourcing, repeated measurements, etc.). They formalize this by introducing an abstract variable Z (e.g., a vector of m noisy labels) and an aggregation function A: Z → 𝒜 (majority vote, K‑nearest‑neighbor voting, etc.). The learning problem then becomes minimizing the aggregated surrogate risk R_{φ,A}(f)=E