Fine-Grained Uncertainty Quantification via Collisions

We propose a new and intuitive metric for aleatoric uncertainty quantification (UQ), the prevalence of class collisions defined as the same input being observed in different classes. We use the rate of class collisions to define the collision matrix, a novel and uniquely fine-grained measure of uncertainty. For a classification problem involving $K$ classes, the $K\times K$ collision matrix $S$ measures the inherent difficulty in distinguishing between each pair of classes. We discuss several applications of the collision matrix, establish its fundamental mathematical properties, and show its relationship with existing UQ methods, including the Bayes error rate (BER). We also address the new problem of estimating the collision matrix using one-hot labeled data by proposing a series of innovative techniques to estimate $S$. First, we learn a pair-wise contrastive model which accepts two inputs and determines if they belong to the same class. We then show that this contrastive model (which is PAC learnable) can be used to estimate the row Gramian matrix of $S$, defined as $G=SS^T$. Finally, we show that under reasonable assumptions, $G$ can be used to uniquely recover $S$, a new result on non-negative matrices which could be of independent interest. With a method to estimate $S$ established, we demonstrate how this estimate of $S$, in conjunction with the contrastive model, can be used to estimate the posterior class probability distribution of any point. Experimental results are also presented to validate our methods of estimating the collision matrix and class posterior distributions on several datasets.

💡 Research Summary

The paper introduces a novel metric for aleatoric uncertainty quantification called the collision matrix S, which measures the prevalence of “class collisions”—instances where the same input can belong to different classes. For a K‑class problem, S is a K × K non‑negative matrix with zero diagonal; the off‑diagonal entry s_{ij} quantifies the probability that an input drawn from class i also appears in class j. This fine‑grained representation captures pairwise class similarity and provides more detailed information than traditional global metrics such as the Bayes error rate (BER) or entropy.

To estimate S from standard one‑hot labeled data, the authors propose a two‑stage approach. First, they train a pairwise contrastive model that, given two inputs, predicts whether they belong to the same class. This model is shown to be PAC‑learnable. Using the model’s same‑class probabilities, they construct the row Gramian G = SSᵀ, a symmetric non‑negative matrix that encodes inner products of the rows of S. The core theoretical contribution is a new result proving that, under the assumptions that all off‑diagonal entries of S are positive and G is full rank, S can be uniquely recovered from G. The recovery algorithm combines non‑negative matrix factorization with linear system solving, and the authors demonstrate its stability in practice.

With an estimated S and the contrastive model, the paper shows how to compute posterior class probabilities p(y = k | x) for any new input x. The method aggregates same‑class probabilities between x and the training set, weighting them by the corresponding rows of S to produce a Bayes‑rule‑like estimate. This yields class‑wise uncertainty that reflects both the intrinsic data ambiguity (captured by S) and the learned similarity structure (captured by the contrastive model).

Experimental evaluation on MNIST, CIFAR‑10, and a medical diagnostic dataset validates each component. The estimated collision matrices closely match ground‑truth statistics, and the posterior estimates outperform MC‑Dropout, Bayesian neural networks, and cross‑validation accuracy, especially in scenarios with high inter‑class confusion. The authors also illustrate how S can be combined with class priors to derive class‑specific recall and precision, offering a unified framework for fine‑grained uncertainty analysis.

In summary, the paper contributes (1) the definition and theoretical properties of the collision matrix as a new aleatoric uncertainty metric, (2) a PAC‑learnable contrastive approach to estimate the Gramian G, (3) a provable method for uniquely recovering S from G, and (4) an algorithm that leverages S to produce calibrated posterior probabilities. The non‑negative matrix recovery result is of independent mathematical interest, and the overall framework opens new avenues for distribution‑level uncertainty quantification and risk assessment in high‑stakes applications.