Evaluating Prediction-based Interventions with Human Decision Makers In Mind

Automated decision systems (ADS) are broadly deployed to inform and support human decision-making across a wide range of consequential settings. However, various context-specific details complicate the goal of establishing meaningful experimental evaluations for prediction-based interventions. Notably, current experiment designs rely on simplifying assumptions about human decision making in order to derive causal estimates. In reality, specific experimental design decisions may induce cognitive biases in human decision makers, which could then significantly alter the observed effect sizes of the prediction intervention. In this paper, we formalize and investigate various models of human decision-making in the presence of a predictive model aid. We show that each of these behavioural models produces dependencies across decision subjects and results in the violation of existing assumptions, with consequences for treatment effect estimation. This work aims to further advance the scientific validity of intervention-based evaluation schemes for the assessment of ADS deployments.

💡 Research Summary

**
The paper addresses a critical gap in the evaluation of prediction‑based interventions (PBIs) that are deployed alongside human decision‑makers. While automated decision systems (ADS) are increasingly used in high‑stakes domains such as criminal justice, healthcare, and education, most experimental evaluations assume that human actors simply receive a treatment indicator (e.g., “algorithm shown” vs. “algorithm hidden”) and then produce an outcome. This simplifying assumption ignores the cognitive processes by which humans interpret, trust, or reject algorithmic predictions. The authors formalize several behavioral models of human decision‑making in the presence of a predictive aid—confirmation bias, anchoring, and information‑overload avoidance—and embed them in a causal graphical model that includes latent variables for individual cognitive bias (J_i) and random noise (ε_i).

A key insight is that these behavioral models generate dependencies across decision subjects, violating the Stable Unit Treatment Value Assumption (SUTVA) that underlies most causal estimators. In particular, even when the treatment assignment Z (whether the algorithm is shown) is randomized, the human’s response D_i can be a function of the algorithm’s prediction Ŷ_i, which itself depends on the treatment. This creates a non‑causal pathway from Z to D_i that interferes with other subjects, leading to biased estimates of the average treatment effect (ATE).

The authors identify three experimental design choices that shape this interference: (1) the binary treatment assignment Z, (2) the model’s positive‑prediction rate P(Ŷ=1) (controlled by the decision threshold), and (3) the model’s overall accuracy P(Ŷ=Y) (determined by model selection). They show analytically how each choice influences “judge responsiveness” – the probability that a human will follow the algorithmic recommendation – and consequently how it biases the ATE.

To illustrate the practical impact, the paper re‑analyses data from Imai et al. (2020), a randomized controlled trial (RCT) that evaluated a public‑safety risk‑score. By systematically varying the positive‑prediction rate (e.g., from 30 % to 70 %) and the model accuracy (e.g., from 0.6 to 0.9), the authors simulate human responses under the different behavioral models. The simulations reveal that ATE estimates can shift by 0.12–0.18 points purely due to design‑induced cognitive bias, a magnitude that could materially affect policy decisions. Moreover, incorporating the latent bias variables into a Bayesian hierarchical model reduces the bias and yields tighter credible intervals for the treatment effect.

The paper’s contributions are threefold: (1) a formal taxonomy of human‑algorithm interaction models that expose hidden dependencies; (2) a causal‑graphical framework that makes explicit where standard SUTVA assumptions break down; (3) empirical evidence that modest changes in experimental design can substantially alter estimated treatment effects.

In the discussion, the authors recommend that future ADS evaluations (a) conduct pre‑studies to characterize the specific cognitive biases of the target decision‑makers, (b) carefully choose algorithmic thresholds and model families to minimize unintended bias amplification, and (c) employ diagnostic checks for interference (e.g., testing for cross‑subject correlation in outcomes). They also suggest extending the methodology to field experiments where latent bias variables can be measured directly (e.g., via surveys or psychometric tests) and to multi‑arm designs that simultaneously test several algorithmic configurations.

Overall, the paper convincingly argues that evaluating prediction‑based interventions without accounting for human cognitive behavior leads to scientifically invalid conclusions. By integrating behavioral modeling into causal inference, it provides a roadmap for more reliable, ethically sound, and policy‑relevant assessments of automated decision‑making systems.