Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators

In the era of Model-as-a-Service, organizations increasingly rely on third-party AI models for rapid deployment. However, the dynamic nature of emerging AI applications, the continual introduction of new datasets, and the growing number of models claiming superior performance make efficient and reliable validation of model services increasingly challenging. This motivates the development of sample-efficient performance estimators, which aim to estimate model performance by strategically selecting instances for labeling, thereby reducing annotation cost. Yet existing evaluation approaches often fail in low-variance settings: RMSE conflates bias and variance, masking persistent bias when variance is small, while p-value based tests become hypersensitive, rejecting adequate estimators for negligible deviations. To address this, we propose a fault-tolerant evaluation framework that integrates bias and variance considerations within an adjustable tolerance level ${\varepsilon}$, enabling the evaluation of performance estimators within practically acceptable error margins. We theoretically show that proper calibration of ${\varepsilon}$ ensures reliable evaluation across different variance regimes, and we further propose an algorithm that automatically optimizes and selects ${\varepsilon}$. Experiments on real-world datasets demonstrate that our framework provides comprehensive and actionable insights into estimator behavior.

💡 Research Summary

In the rapidly expanding Model‑as‑a‑Service (MaaS) landscape, organizations increasingly rely on third‑party AI models accessed via APIs for downstream decision‑making. Because labeling large test sets is costly and data distributions evolve, practitioners turn to sample‑efficient performance estimators that strategically select a small subset of instances for annotation in order to approximate a model’s true performance (θ*). The dominant evaluation practices for such estimators are root‑mean‑square error (RMSE) and two‑sided t‑tests that produce p‑values. This paper demonstrates that both metrics become unreliable in low‑variance regimes—a situation that naturally arises when the labeling budget grows or when advanced estimators such as Active Testing (AT) aggressively reduce variance.

RMSE conflates bias² and variance into a single scalar. As variance shrinks, RMSE can remain low even if a systematic bias persists or worsens, leading to the paradox that a biased estimator appears “better” simply because it uses more labels. Conversely, the traditional two‑sided t‑test evaluates the ratio of bias to variability; when variance is tiny, even a minute bias yields a large t‑statistic and a tiny p‑value, causing practitioners to reject an estimator that is practically acceptable. Empirical illustrations on 20 Newsgroup, CIFAR‑10, and IMDB datasets show that AT’s mean estimate drifts upward while its RMSE stays flat, and its p‑value becomes highly significant despite the drift being well within a reasonable error margin.

To overcome these shortcomings, the authors propose a Fault‑Tolerant Evaluation framework (FT‑Eval). The core idea is to introduce a user‑defined tolerance ε that reflects an acceptable error band around the ground truth. FT‑Eval employs two one‑sided tests (TOST) to assess whether the estimator’s output lies within