Label-Efficient Monitoring of Classification Models via Stratified Importance Sampling

Monitoring the performance of classification models in production is critical yet challenging due to strict labeling budgets, one-shot batch acquisition of labels and extremely low error rates. We propose a general framework based on Stratified Importance Sampling (SIS) that directly addresses these constraints in model monitoring. While SIS has previously been applied in specialized domains, our theoretical analysis establishes its broad applicability to the monitoring of classification models. Under mild conditions, SIS yields unbiased estimators with strict finite-sample mean squared error (MSE) improvements over both importance sampling (IS) and stratified random sampling (SRS). The framework does not rely on optimally defined proposal distributions or strata: even with noisy proxies and sub-optimal stratification, SIS can improve estimator efficiency compared to IS or SRS individually, though extreme proposal mismatch may limit these gains. Experiments across binary and multiclass tasks demonstrate consistent efficiency improvements under fixed label budgets, underscoring SIS as a principled, label-efficient, and operationally lightweight methodology for post-deployment model monitoring.

💡 Research Summary

The paper tackles the practical problem of monitoring the performance of deployed classification models when the labeling budget is severely limited and the true error rates are extremely low. In such settings, naïve random sampling provides virtually no information, while existing approaches—stratified random sampling (SRS) and importance sampling (IS)—each have drawbacks: SRS depends on a good stratification that may be unavailable, and IS relies on a proposal distribution that can be mismatched with the true data distribution, leading to high variance or bias. Adaptive methods that continuously update sampling probabilities are often infeasible for batch‑wise annotation pipelines.

To address these challenges, the authors propose a general framework based on Stratified Importance Sampling (SIS). The method first partitions the unlabeled data set into P disjoint strata using simple, domain‑agnostic heuristics (e.g., quantiles of continuous features, predicted class and brightness for images). Each stratum j receives a proportion w_j of the total labeling budget, following proportional allocation (n_j ≈ n·w_j). Within each stratum, a proposal distribution q_j is defined by re‑weighting a model‑derived score s(x) (e.g., uncertainty, entropy, confidence) as q(x) ∝ s(x)^α and then normalizing it inside the stratum. Samples are drawn from q_j, the true labels are queried, and an importance‑weighted estimator of the error signal Z = 1{ŷ ≠ y} is computed. The overall SIS estimator is a weighted sum of the stratum‑specific IS estimators.

The theoretical contribution consists of two theorems that compare the mean‑squared error (MSE) of SIS against IS and SRS under mild assumptions (absolute continuity p_j ≪ q_j, finite second moments, proportional allocation). Theorem 1 decomposes the variance reduction of SIS over IS into (i) a “proposal mismatch” term that captures how the global proposal distribution allocates probability across strata relative to the proportional allocation, and (ii) an inter‑stratum variance term reflecting the variability of stratum‑wise error rates. If both terms are non‑positive, SIS strictly dominates IS. Theorem 2 shows that SIS beats SRS whenever the weighted average of the within‑stratum variance gaps Δ_j(q_j) is negative; in other words, strong variance reductions in some strata can compensate for weaker reductions (or even increases) in others. These results guarantee that even with noisy proxies and sub‑optimal strata, SIS will typically achieve lower MSE than either baseline, unless the proposal distribution is extremely poorly aligned.

Empirically, the authors evaluate SIS on six datasets covering binary and multiclass tasks, tabular and image modalities, and a wide range of defect rates (from <1 % to >18 %). For each dataset they simulate a batch‑wise monitoring scenario with a fixed labeling budget and compare seven sampling designs: random sampling (RS), SRS, IS, FILA, an adaptive SRS variant, an adaptive IS variant, and the proposed SIS. Strata are constructed from simple binning of features or model scores, and the proposal distribution is tuned via a scalar α that controls its sharpness. Results consistently show that SIS attains the lowest MSE across all datasets, with especially large gains on low‑error image tasks (MNIST, CIFAR‑10) where naïve sampling would require orders of magnitude more labels to achieve comparable accuracy. The method also proves robust to the choice of α; even when the proposal distribution is only loosely correlated with the true error probability, SIS still outperforms the baselines.

The paper discusses limitations: (1) if the proposal distribution is severely mismatched, the first term in Theorem 1 can become positive, eroding the advantage over IS; (2) overly fine stratification can lead to strata with too few allocated samples, inflating variance. Both issues can be mitigated in practice by merging tiny strata and performing a modest grid search over α. The authors also note that while the current analysis focuses on estimating the 0‑1 error rate, the SIS framework can be extended to other performance metrics (e.g., AUC, F1) with additional theoretical work.

In conclusion, the work provides a statistically sound, computationally lightweight, and easily deployable solution for label‑efficient model monitoring. By jointly leveraging stratification and importance weighting, SIS delivers finite‑sample MSE guarantees that surpass traditional IS and SRS, without requiring continuous adaptation or extensive labeled data. This makes it a compelling tool for high‑stakes applications such as medical diagnosis, fraud detection, and safety‑critical image classification, where monitoring must be both accurate and economical.