Singleton-Optimized Conformal Prediction

Conformal prediction can be used to construct prediction sets that cover the true outcome with a desired probability, but can sometimes lead to large prediction sets that are costly in practice. The most useful outcome is a singleton prediction-an unambiguous decision-yet existing efficiency-oriented methods primarily optimize average set size. Motivated by this, we propose a new nonconformity score that aims to minimize the probability of producing non-singleton sets. Starting from a non-convex constrained optimization problem as a motivation, we provide a geometric reformulation and associated algorithm for computing the nonconformity score and associated split conformal prediction sets in O(K) time for K-class problems. Using this score in split conformal prediction leads to our proposed Singleton-Optimized Conformal Prediction (SOCOP) method. We evaluate our method in experiments on image classification and LLM multiple-choice question-answering, comparing with standard nonconformity scores such as the (negative) label probability estimates and their cumulative distribution function; both of which are motivated by optimizing length. The results show that SOCOP increases singleton frequency (sometimes by over 20%) compared to the above scores, with minimal impact on average set size.

💡 Research Summary

The paper introduces a novel conformal prediction framework called Singleton‑Optimized Conformal Prediction (SOCOP) that explicitly targets the production of singleton prediction sets—sets containing exactly one label—rather than merely minimizing the expected size of the prediction set. Traditional conformal methods focus on guaranteeing marginal coverage while reducing the average cardinality of the set, which can still be costly when the set size exceeds one because downstream processes often require additional human verification or workflow changes. The authors therefore adopt the M‑criterion originally described by Vovk et al. (2005), which seeks to minimize the probability of producing a non‑singleton set, and they integrate this objective with a regularization term that controls the trade‑off between singleton frequency and average set length.

Problem Formulation

The authors formalize the goal as a constrained optimization problem:

\