Reevaluating Causal Estimation Methods with Data from a Product Release

Recent developments in causal machine learning methods have made it easier to estimate flexible relationships between confounders, treatments and outcomes, making unconfoundedness assumptions in causal analysis more palatable. How successful are these approaches in recovering ground truth baselines? In this paper we analyze a new data sample including an experimental rollout of a new feature at a large technology company and a simultaneous sample of users who endogenously opted into the feature. We find that recovering ground truth causal effects is feasible – but only with careful modeling choices. Our results build on the observational causal literature beginning with LaLonde (1986), offering best practices for more credible treatment effect estimation in modern, high-dimensional datasets.

💡 Research Summary

This paper provides a rigorous, real‑world validation of modern causal‑inference methods by exploiting a paired dataset released by Microsoft. The dataset consists of two closely matched samples covering the same two‑week period in December 2022: (1) an experimental sample in which a new Windows feature was randomly turned on for a subset of devices, and (2) an observational sample in which users could voluntarily enable the same feature. Both samples contain a rich set of over 200 covariates describing device specifications, user behavior, and anonymized geographic information, making the overlap between treated and control units far richer than classic benchmarks such as LaLonde (1986).

The authors focus on two performance outcomes—one continuous and one binary—and aim to recover the average treatment effect (ATE) as well as conditional average treatment effects (CATE) from the observational data, comparing the estimates against the “gold‑standard” ATE obtained from the randomized experiment. They evaluate a spectrum of estimators: simple difference‑in‑means, linear regression, propensity‑score matching, flexible machine‑learning propensity models (random forests, LASSO, gradient boosting), doubly‑robust (DR) estimators, and targeted maximum likelihood estimation (TMLE).

Key methodological steps include:

1. Overlap assessment and optimal trimming – using the Crump et al. (2009) rule to drop units with extreme estimated propensity scores (e(x) < α or > 1 − α), thereby ensuring the positivity assumption and reducing variance.
2. Sample splitting and hyper‑parameter tuning – each machine‑learning model is trained on a training fold and evaluated on a hold‑out fold; hyper‑parameters are chosen by cross‑validation to avoid over‑fitting.
3. Doubly‑robust and TMLE implementation – both the outcome regression and the propensity model are allowed to be flexible; consistency is retained as long as at least one of the two nuisance models is correctly specified.
4. Model averaging – the authors average across several DR specifications to mitigate model‑selection uncertainty, following Breiman’s (1996) “bagging” principle.

Findings for the continuous outcome: After trimming and employing a well‑tuned DR estimator, the observational estimate of the ATE (≈ 0.118) falls squarely within the 95 % confidence interval of the experimental ATE (≈ 0.12). This demonstrates that, with sufficient covariate richness and disciplined modeling, observational data can faithfully reproduce the causal effect measured in a randomized trial.

Findings for the binary outcome: All observational estimators over‑state the negative effect (≈ ‑0.54) relative to the experimental benchmark (≈ ‑0.43). Sensitivity analysis using Chernozhukov et al. (2022) Rosenbaum bounds shows that an unobserved confounder would need an implausibly large impact to drive the estimate to zero, indicating that the sign of the effect is robust, but its magnitude remains biased. The authors attribute the residual bias to (i) unmeasured factors (e.g., user satisfaction, network conditions) and (ii) possible nonlinear interactions not captured by the models.

Heterogeneity analysis: Using the DR‑score test (Chernozhukov et al., 2024), the authors detect meaningful CATE variation for the binary outcome. Devices with higher usage frequency experience larger declines in the binary performance metric, whereas the continuous outcome shows no detectable heterogeneity. This suggests that policy decisions (e.g., targeted roll‑outs) should consider user‑segment characteristics.

Practical recommendations:

• Best‑practice pipeline: (a) verify overlap and trim extreme propensity scores, (b) split the sample for out‑of‑sample tuning, (c) employ doubly‑robust estimators with flexible machine‑learning nuisance models, and (d) average across several specifications to reduce model uncertainty.
• Diagnostic checks: Conduct multiple plausibility tests for unconfoundedness (sensitivity analyses, assessment of predictive power of covariates on outcomes) because the assumption cannot be directly verified.
• Limitations: For outcomes with strong selection bias and limited predictive covariates (as with the binary metric), observational methods may still leave substantial bias; additional data collection or structural modeling may be required.

Overall, the paper demonstrates that modern causal‑machine‑learning tools can recover true causal effects from high‑dimensional observational data when applied with rigorous preprocessing, careful hyper‑parameter tuning, and robust diagnostics. At the same time, it underscores that the credibility of estimates hinges on the quality of covariates and the thoroughness of validation steps, offering a concrete roadmap for researchers and practitioners seeking credible treatment‑effect estimates in real‑world, high‑dimensional settings.