Randomization Does Not Justify Logistic Regression

The logit model is often used to analyze experimental data. However, randomization does not justify the model, so the usual estimators can be inconsistent. A consistent estimator is proposed. Neyman’s non-parametric setup is used as a benchmark. In this setup, each subject has two potential responses, one if treated and the other if untreated; only one of the two responses can be observed. Beside the mathematics, there are simulation results, a brief review of the literature, and some recommendations for practice.

💡 Research Summary

The paper challenges the widespread practice of applying logistic regression to data from randomized experiments on the assumption that randomization alone validates the model. Using Neyman’s non‑parametric potential‑outcome framework, the author shows that each experimental unit possesses two potential responses—one under treatment and one under control—and only one is observed. Random assignment guarantees independence between treatment assignment and the potential outcomes, but it does not impose any functional form on the relationship between covariates and the binary response. Logistic regression, however, assumes a specific structural form: the log‑odds of success is a linear function of covariates, implying that the treatment effect is a constant odds ratio across all units. This is a strong assumption about the distribution of the potential outcomes, and it is not ensured by randomization.

The paper proves that when the true data‑generating process violates the logistic form—e.g., when treatment effects are heterogeneous or the baseline risk follows a non‑logistic curve—the maximum‑likelihood estimator (MLE) of the logistic coefficients can be biased and inconsistent. In contrast, a simple non‑parametric estimator of the average treatment effect (ATE), defined as the difference between the observed success rates in the treated and control groups, remains consistent regardless of the underlying potential‑outcome distribution. The author derives the asymptotic normality of this estimator, provides a robust variance estimator, and shows how to construct valid confidence intervals.

A comprehensive simulation study illustrates these points. Two scenarios are considered: (1) data generated exactly according to a logistic model, and (2) data generated from a model that departs substantially from logistic assumptions (e.g., a quadratic log‑odds function or unit‑specific odds ratios). For each scenario, 1,000 Monte‑Carlo repetitions are run with varying sample sizes. Results indicate that while logistic regression performs efficiently when its assumptions hold, it exhibits large bias, inflated mean‑squared error, and under‑coverage of nominal 95 % confidence intervals in the misspecified scenario. The proposed non‑parametric ATE estimator, by contrast, shows negligible bias and appropriate coverage in both scenarios, confirming its robustness.

The literature review situates the contribution within a broader debate on causal inference. The author references classic works by Neyman (1923), Rubin (1974), and modern texts by Imbens & Rubin (2015), emphasizing that randomization is a prerequisite for unbiased causal estimation but does not replace model checking. Prior studies that have advocated logistic regression for randomized trials often conflate predictive modeling with causal effect estimation, leading to misinterpretation of odds ratios as causal quantities.

Based on the theoretical and empirical findings, the paper offers concrete recommendations for practitioners: (1) before fitting a logistic model, explicitly assess whether the constant‑odds assumption is plausible; (2) use the simple difference‑in‑proportions estimator as a baseline analysis for the average treatment effect; (3) if logistic regression is still desired for its interpretability or for adjusting covariates, perform post‑hoc goodness‑of‑fit tests (e.g., Hosmer‑Lemeshow, calibration plots) and consider flexible extensions such as generalized additive models; (4) employ robust standard errors or bootstrap methods, especially in moderate sample sizes; and (5) report both the model‑based odds ratios and the non‑parametric ATE to provide a fuller picture of the causal effect.

In conclusion, the paper asserts that randomization alone does not justify the logistic regression model. Consistent causal inference in randomized experiments requires either a verification of the logistic structural assumptions or the adoption of non‑parametric estimators that are immune to model misspecification. This insight has important implications for the analysis of clinical trials, field experiments, and any setting where binary outcomes are examined under random assignment.