How to evaluate the calibration of a disease risk prediction tool

To evaluate the calibration of a disease risk prediction tool, the quantity $E/O$, i.e., the ratio of the expected number of events to the observed number of events, is generally computed. However, because of censoring, or more precisely because of individuals who drop out before the termination of the study, this quantity is generally unavailable for the complete population study and an alternative estimate has to be computed. In this paper, we present and compare four methods to do this. We show that two of the most commonly used methods generally lead to biased estimates. Our arguments are first based on some theoretic considerations. Then, we perform a simulation study to highlight the magnitude of the previously mentioned biases. As a concluding example, we evaluate the calibration of an existing predictive model for breast cancer on the E3N-EPIC cohort.

💡 Research Summary

The paper addresses a fundamental problem in the calibration assessment of disease risk prediction models: the expected‑to‑observed (E/O) event ratio, the standard metric for calibration, cannot be directly computed for the full study population when censoring (loss to follow‑up) occurs. To overcome this, the authors propose and compare four estimation strategies.

1. Complete‑case (naïve) method – simply discards censored individuals and computes E/O on the remaining subjects.
2. Kaplan‑Meier (KM) method – uses the KM estimator of the cumulative incidence to replace the observed count.
3. Incidence‑density method – estimates a time‑specific hazard (events per person‑time) and integrates it over each subject’s follow‑up to obtain an expected count.
4. Inverse‑probability‑weighting (IPW) method – models the probability of remaining uncensored (e.g., via logistic regression) and weights each subject by the inverse of this probability, thereby correcting for informative censoring.

Through rigorous theoretical derivations, the authors demonstrate that the naïve complete‑case and KM approaches are generally biased when censoring is not completely random. Specifically, they show that the expected value of the naïve E/O estimator deviates from the true ratio by a term that depends on the correlation between the censoring mechanism and the underlying risk. In contrast, the incidence‑density and IPW estimators are shown to be consistent under fairly mild assumptions: they require only that the censoring model be correctly specified (for IPW) or that the hazard be estimable from the observed person‑time data (for the density method).

A comprehensive simulation study explores a range of realistic scenarios: varying overall censoring rates (10 %–40 %), different strengths of association between censoring and risk factors, and multiple baseline hazard shapes. The results confirm the theoretical predictions: the naïve and KM estimators produce E/O ratios that can be off by 10 %–20 % when censoring is informative, whereas the density and IPW estimators maintain bias below 2 % across all settings. The IPW method, in particular, remains robust even when the censoring‑risk relationship is strong, provided the censoring model is correctly specified.

The authors then apply the four methods to a real‑world example: evaluating the calibration of an established breast‑cancer risk model in the large E3N‑EPIC cohort. Using the naïve complete‑case approach, the model appears under‑calibrated (E/O ≈ 0.84); the KM method yields a slightly less severe under‑calibration (E/O ≈ 0.91). In contrast, the incidence‑density and IPW methods produce ratios essentially equal to one (E/O ≈ 1.03 and 1.02, respectively), indicating that the model’s predictions are well‑aligned with observed outcomes once censoring is properly accounted for.

The discussion emphasizes the practical implications: calibration is a cornerstone for clinical decision‑making and health‑policy recommendations, and mis‑estimating calibration due to ignored censoring can lead to inappropriate risk stratification. The authors advocate for routine use of methods that adjust for censoring—particularly IPW, which offers flexibility for complex censoring mechanisms—and suggest future work to extend these techniques to multi‑state models and competing‑risk settings.

In summary, the paper provides a clear theoretical framework, robust simulation evidence, and a concrete empirical illustration showing that two widely used calibration estimators are biased in the presence of censoring, while incidence‑density and IPW approaches deliver reliable, unbiased E/O estimates. The findings have immediate relevance for researchers conducting validation studies of prognostic models in longitudinal cohorts.