Selective Prior Synchronization via SYNC Loss

Prediction under uncertainty is a critical requirement for the deep neural network to succeed responsibly. This paper focuses on selective prediction, which allows DNNs to make informed decisions about when to predict or abstain based on the uncertainty level of their predictions. Current methods are either ad-hoc such as SelectiveNet, focusing on how to modify the network architecture or objective function, or post-hoc such as softmax response, achieving selective prediction through analyzing the model’s probabilistic outputs. We observe that post-hoc methods implicitly generate uncertainty information, termed the selective prior, which has traditionally been used only during inference. We argue that the selective prior provided by the selection mechanism is equally vital during the training stage. Therefore, we propose the SYNC loss which introduces a novel integration of ad-hoc and post-hoc method. Specifically, our approach incorporates the softmax response into the training process of SelectiveNet, enhancing its selective prediction capabilities by examining the selective prior. Evaluated across various datasets, including CIFAR-100, ImageNet-100, and Stanford Cars, our method not only enhances the model’s generalization capabilities but also surpasses previous works in selective prediction performance, and sets new benchmarks for state-of-the-art performance.

💡 Research Summary

**
The paper tackles the problem of selective prediction, where a deep neural network must decide whether to output a prediction or abstain based on its confidence. Existing approaches fall into two categories. “Ad‑hoc” methods such as SelectiveNet (SN) and Deep Gamblers (DG) modify the network architecture or training objective to learn a dedicated selection head that produces a confidence score. These methods, however, rely on the training loss to guide the selection head, which can lead to over‑fitting on the training set and poor discrimination of truly hard examples, especially at low coverage levels. “Post‑hoc” methods like Softmax Response (SR) and MC‑Dropout do not alter the model; they infer confidence from the softmax probabilities or from multiple stochastic forward passes. The authors observe that SR implicitly provides an uncertainty estimate they term the “selective prior,” which is traditionally used only at inference time.

The central contribution is the SYNC loss, a new regularization term that aligns the selection head’s output with the selective prior during training. Formally, let g(x) be the selection score produced by the SN selection head, and let p(x) be the softmax probability vector of the prediction head. The SYNC loss is defined as ℓ_sync(g(x), p(x)) = ℓ(g(x), score(p(x))), where ℓ is a standard regression loss (e.g., mean‑squared error) and score(p(x)) is an uncertainty estimator derived from p(x). In the basic implementation score(p(x)) = max_i p_i(x), i.e., the Softmax Response. The overall training objective becomes

L_SYNC = R(f,g|S) + λ ℓ(c, E