Anytime-Valid Conformal Risk Control

Prediction sets provide a means of quantifying the uncertainty in predictive tasks. Using held out calibration data, conformal prediction and risk control can produce prediction sets that exhibit statistically valid error control in a computationally efficient manner. However, in the standard formulations, the error is only controlled on average over many possible calibration datasets of fixed size. In this paper, we extend the control to remain valid with high probability over a cumulatively growing calibration dataset at any time point. We derive such guarantees using quantile-based arguments and illustrate the applicability of the proposed framework to settings involving distribution shift. We further establish a matching lower bound and show that our guarantees are asymptotically tight. Finally, we demonstrate the practical performance of our methods through both simulations and real-world numerical examples.

💡 Research Summary

This paper addresses a fundamental limitation of standard split‑conformal prediction and conformal risk control: the guarantee that the miscoverage (or more generally, the risk) of a prediction set is bounded by a user‑specified level α holds only on average over all possible calibration sets of a fixed size. In practice, for a single realized calibration set the actual risk can deviate substantially from α, and the probability of such deviation does not vanish even for moderately large calibration sizes (e.g., with n = 500 there is a 48 % chance that the miscoverage exceeds α). Moreover, many real‑world applications collect data sequentially, so the calibration set grows over time and a whole sequence of prediction sets ({C_n(X)}_{n=1}^\infty) is produced. The classical guarantee (1) controls the risk of each individual set in isolation, but it does not ensure that all sets in the sequence simultaneously respect the risk budget with high probability.

The authors introduce the notion of anytime‑valid risk control (Definition 2.7). A method is anytime‑valid if, with probability at least (1-\delta), every conditional risk (E