Winner's Curse Drives False Promises in Data-Driven Decisions: A Case Study in Refugee Matching

A major challenge in data-driven decision-making is accurate policy evaluation-i.e., guaranteeing that a learned decision-making policy achieves the promised benefits. A popular strategy is model-based policy evaluation, which estimates a model from data to infer counterfactual outcomes. This strategy is known to produce unwarrantedly optimistic estimates of the true benefit due to the winner’s curse. We searched the recent literature on data-driven decision-making, identifying a sample of 55 papers published in the Management Science in the past decade; all but two relied on this flawed methodology. Several common justifications are provided: (1) the estimated models are accurate, stable, and well-calibrated, (2) the historical data uses random treatment assignment, (3) the model family is well-specified, and (4) the evaluation methodology uses sample splitting. Unfortunately, we show that no combination of these justifications avoids the winner’s curse. First, we provide a theoretical analysis demonstrating that the winner’s curse can cause large, spurious reported benefits even when all these justifications hold. Second, we perform a simulation study based on the recent and consequential data-driven refugee matching problem. We construct a synthetic refugee matching environment (calibrated to closely match the real setting) but designed so that no assignment policy can improve expected employment compared to random assignment. Model-based methods report large, stable gains of around 60% even when the true effect is zero; these gains are on par with improvements of 22-75% reported in the literature. Our results provide strong evidence against model-based evaluation.

💡 Research Summary

The paper tackles a fundamental problem in data‑driven decision making: how to reliably evaluate whether a learned policy truly delivers the promised benefits. Two broad families of evaluation methods exist. Model‑free approaches, such as inverse‑probability weighting (IPW), rely on random treatment assignment and can provide unbiased estimates of policy performance, but they suffer from extreme variance when the action space is large or covariates are high‑dimensional. Consequently, many researchers resort to model‑based evaluation, which first fits a predictive model of outcomes (and possibly treatment effects) on historical data and then uses this model to impute counterfactual outcomes for any candidate policy.

The authors surveyed 55 recent “estimate‑then‑optimize” papers published in Management Science over the past decade; 53 (96 %) employed model‑based evaluation. The papers typically justified this choice with four arguments: (J1) the model is accurate, stable and well‑calibrated; (J2) the historical data were generated by random assignment; (J3) the model class is sufficiently rich to contain the true data‑generating process; and (J4) evaluation uses a separate sample (sample splitting) from the one used for policy learning.

The central claim of the paper is that none of these justifications shield model‑based evaluation from the “winner’s curse.” The winner’s curse arises because the same estimated model is used both to select a policy (optimizing over predicted outcomes) and to evaluate that policy. Optimization exploits random estimation errors, preferentially picking actions whose predicted benefit exceeds the true benefit. When the same biased predictions are then used for evaluation, the bias remains uncorrected, leading to systematically optimistic performance estimates.

To demonstrate the pervasiveness of this phenomenon, the authors provide two stylized theoretical constructions. First, they consider a misspecified parametric model (e.g., a linear model applied to a fundamentally nonlinear relationship). Even if the model achieves high predictive accuracy on the training distribution, the misspecification induces a systematic positive bias in counterfactual predictions under the policy‑induced distribution. Consequently, the estimated policy improvement can be arbitrarily large despite J1, J2 and J4 holding. Second, they examine regularized non‑parametric learners (random forests, gradient‑boosted trees, ridge regression). Regularization introduces a small bias on the training distribution, but this bias can be amplified when the policy shifts the covariate distribution, again yielding large positive bias even when all four justifications are satisfied.

Empirically, the paper focuses on the refugee‑matching problem, a high‑impact application where researchers have claimed 22 %–75 % employment gains from data‑driven placement algorithms (Bansak et al., 2018; Ahani et al., 2021). The authors construct a synthetic environment calibrated to the real data but deliberately designed so that no policy can outperform random assignment; the true policy effect is exactly zero. Applying the exact model‑based evaluation pipeline from the original studies to this synthetic data, they obtain spurious estimated gains of roughly 60 %—comparable to the gains reported in the literature. A bootstrap‑based variant of the evaluation also reports stable, large improvements despite the null ground truth.

These results lead to several important conclusions. First, the winner’s curse is not a fringe issue limited to poorly performing or unstable models; it can arise even when models are accurate, the data are randomly assigned, the model class is correctly specified, and sample splitting is employed. Second, the prevalence of model‑based evaluation in top‑tier management science literature means that many published policy‑improvement claims may be substantially overstated. Third, the authors advocate for a shift toward more robust evaluation methods: (i) refined IPW or doubly‑robust estimators that control variance; (ii) restricting the policy class to reduce selection bias; (iii) integrating auxiliary real‑world outcome data to anchor counterfactual estimates; and (iv) designing learning objectives that target statistical significance rather than raw expected gain.

In sum, the paper provides a rigorous theoretical proof and a realistic simulation that together demonstrate the inevitability of the winner’s curse under common modeling practices. It calls for a re‑examination of evaluation standards in data‑driven policy research and for the development of methods that can deliver trustworthy performance guarantees without succumbing to optimistic bias.