Optimal training-conditional regret for online conformal prediction

We study online conformal prediction for non-stationary data streams subject to unknown distribution drift. While most prior work studied this problem under adversarial settings and/or assessed performance in terms of gaps of time-averaged marginal coverage, we instead evaluate performance through training-conditional cumulative regret. We specifically focus on independently generated data with two types of distribution shift: abrupt change points and smooth drift. When non-conformity score functions are pretrained on an independent dataset, we propose a split-conformal style algorithm that leverages drift detection to adaptively update calibration sets, which provably achieves minimax-optimal regret. When non-conformity scores are instead trained online, we develop a full-conformal style algorithm that again incorporates drift detection to handle non-stationarity; this approach relies on stability - rather than permutation symmetry - of the model-fitting algorithm, which is often better suited to online learning under evolving environments. We establish non-asymptotic regret guarantees for our online full conformal algorithm, which match the minimax lower bound under appropriate restrictions on the prediction sets. Numerical experiments corroborate our theoretical findings.

💡 Research Summary

This paper addresses the problem of online conformal prediction under non‑stationary data streams where the underlying distribution may drift over time. While most prior work on online conformal inference has focused on either guaranteeing long‑run marginal coverage or on adversarial regret bounds, the authors propose a different performance metric: training‑conditional cumulative regret. This metric measures, at each time step, the deviation of the conditional coverage probability (conditioned on all past observations) from the target level 1 − α, and then aggregates these deviations over time. By aligning the evaluation with the classical notion of conditional coverage, the metric directly reflects the usefulness of the prediction sets at every individual time point.

The authors consider two realistic models of distribution shift for independently generated data: (i) a change‑point model where the distribution is piecewise stationary with a finite number K of abrupt changes, and (ii) a smooth‑drift model where the distribution evolves continuously but its total variation over the horizon is bounded by V. In both settings the learner has no prior knowledge of when or how the drift occurs.

Two algorithmic families are developed, each tailored to a different way of obtaining the non‑conformity score function sₜ(·,·).

1. Pre‑trained scores (split‑conformal style).
   When the score function is trained offline on an independent dataset, the authors propose DriftOCP (Algorithm 2). The method maintains a calibration set drawn from the recent past and updates it adaptively using a drift‑detection subroutine (e.g., CUSUM, Page‑Hinkley, or KL‑based tests). Whenever a change is detected, the calibration set is refreshed with the most recent observations, ensuring that the quantile estimate qₜ used to construct the prediction set Cₜ(x)= {y : sₜ(x,y) ≤ qₜ} reflects the current distribution. DriftOCP is computationally light (O(1) per round) and horizon‑independent. The authors prove non‑asymptotic regret bounds:

   • For the change‑point model, R_T ≤ C √(K T) log T.
   • For the smooth‑drift model, R_T ≤ C √(V T) log T.

   These upper bounds match the minimax lower bounds for the respective settings up to a logarithmic factor, establishing that DriftOCP is essentially optimal.

2. Online‑trained scores (full‑conformal style).
   When the score and the underlying predictive model are updated online, the authors move away from the usual permutation‑symmetry assumption (which rarely holds for streaming learners) and instead assume algorithmic stability: a single update changes the model parameters only by a bounded amount (e.g., Lipschitz‑continuous updates as in online SGD with a small learning rate). Under this stability condition, they introduce DriftOCP‑full (Algorithm 4). The algorithm computes the full‑conformal quantile using all past data, incorporates drift detection to decide when to reset the calibration window, and updates the model incrementally. The stability assumption enables concentration arguments that lead to regret guarantees identical in form to those for the split‑conformal case:

   • Change‑point: R_T ≤ C √(K T) log T.
   • Smooth‑drift: R_T ≤ C √(V T) log T.

   Moreover, the paper derives a universal lower bound on training‑conditional cumulative regret that applies to any prediction‑set construction, showing that no method can beat the √(K T) or √(V T) rate (up to constants). Hence DriftOCP‑full attains minimax optimality under the stability framework.

The theoretical contributions are complemented by extensive experiments. Synthetic data experiments simulate both abrupt change‑points and smooth drifts, demonstrating that DriftOCP and DriftOCP‑full achieve substantially lower cumulative regret than baselines such as Adaptive Conformal Inference (ACI), RAPS, and other recent online conformal methods, while maintaining the prescribed coverage level. Real‑world time‑series (electricity demand, stock prices, web traffic) are also used, with artificially injected drifts to test adaptability. The proposed methods quickly adjust the width of prediction sets after a drift, avoiding the overly conservative intervals that plague non‑adaptive baselines. Sensitivity analyses explore the impact of drift‑detection thresholds and calibration‑window sizes, showing that the algorithms are robust across a wide range of hyper‑parameter choices.

In summary, the paper makes four key contributions:

• Introduces training‑conditional cumulative regret as a principled metric aligned with conditional coverage.
• Designs two adaptive online conformal algorithms (DriftOCP and DriftOCP‑full) that incorporate drift detection and are computationally efficient.
• Provides non‑asymptotic regret bounds that are minimax‑optimal (up to logarithmic factors) for both abrupt and smooth distribution shifts.
• Demonstrates through theory and experiments that the methods reliably achieve target coverage while adapting rapidly to non‑stationarity, even when the underlying model is updated online under stability constraints.

These results open new avenues for reliable uncertainty quantification in streaming environments, suggesting future work on multivariate responses, non‑Euclidean data, and more sophisticated drift‑detection mechanisms.