Validating Causal Message Passing Against Network-Aware Methods on Real Experiments

Estimating total treatment effects in the presence of network interference typically requires knowledge of the underlying interaction structure. However, in many practical settings, network data is either unavailable, incomplete, or measured with substantial error. We demonstrate that causal message passing, a methodology that leverages temporal structure in outcome data rather than network topology, can recover total treatment effects comparable to network-aware approaches. We apply causal message passing to two large-scale field experiments where a recently developed bipartite graph methodology, which requires network knowledge, serves as a benchmark. Despite having no access to the interaction network, causal message passing produces effect estimates that match the network-aware approach in direction across all metrics and in statistical significance for the primary decision metric. Our findings validate the premise of causal message passing: that temporal variation in outcomes can serve as an effective substitute for network observation when estimating spillover effects. This has important practical implications: practitioners facing settings where network data is costly to collect, proprietary, or unreliable can instead exploit the temporal dynamics of their experimental data.

💡 Research Summary

This paper conducts a direct empirical comparison between two fundamentally different approaches for estimating total treatment effects (TTE) in the presence of network interference. The first approach is the bipartite‑graph, network‑aware method introduced by Tan et al. (2025), which explicitly uses the observed bipartite interaction graph between treatment units (e.g., service providers) and connected units (e.g., customers). It constructs direct exposure (E_Dir) and indirect exposure (E_Ind) variables for each treatment unit, fits a flexible outcome model Ψ (typically via kernel ridge regression or other machine learning techniques), and then computes the Primary Total Treatment Effect (PTTE) by contrasting predicted outcomes under the all‑treated and all‑control scenarios. This method assumes that the full interaction network is accurately observed; any missing or noisy edges can induce bias.

The second approach is Causal Message Passing (CMP), a network‑blind methodology that leverages only temporal variation in aggregated outcomes. CMP treats the evolution of the population‑level outcome distribution Y_t and the treatment assignment distribution W_t as a dynamical system: Y_{t+1}=f_t(Y_t,W_{t+1}). The mapping f_t captures both direct and spillover effects without reference to individual edges. CMP proceeds by (1) constructing summary statistics (means, higher moments, interaction terms) for each time period, (2) using supervised learning to estimate the state‑evolution functions f_t from observed (Y_t,W_{t+1}) pairs, and (3) simulating the system under full‑treatment and full‑control policies to obtain an estimate of the total effect. The key theoretical assumption is that the interference mechanism is temporally invariant—its structural rules do not change over the observation window.

The authors evaluate both methods on two large‑scale field experiments conducted in a bipartite setting. Experiment A involves roughly 7,000 eligible treatment units; Experiment B involves about 4,000. Both experiments use a staggered rollout design: all units start in control, and at predetermined dates a subset switches to treatment and remains treated thereafter. Outcomes are aggregated at the treatment‑unit level by summing edge‑level metrics (e.g., service volume, revenue).

Results show that CMP reproduces the direction of the effect estimated by the network‑aware benchmark for every metric examined. For the primary decision metric (the one driving business decisions), CMP’s estimate is not only directionally consistent but also statistically significant at the same level as the bipartite method (p < 0.05). For secondary metrics (customer dwell time, repeat purchase rate, etc.), CMP’s point estimates are very close to those of the benchmark, though confidence intervals are modestly wider, reflecting the additional uncertainty inherent in a network‑blind approach. By contrast, a naïve difference‑in‑means estimator that ignores interference yields biased, often opposite‑signed, results, underscoring the necessity of accounting for spillovers.

The paper’s contributions are threefold: (1) it provides the first real‑world validation of CMP against a state‑of‑the‑art network‑aware estimator, demonstrating comparable accuracy; (2) it shows that temporal dynamics alone can identify the direction of interference bias, effectively correcting naïve estimates without any graph data; (3) it offers practical guidance for practitioners facing costly, proprietary, or unreliable network data, suggesting that CMP is a viable, low‑cost alternative.

Limitations are acknowledged. CMP relies on sufficient temporal variation; very short experiments or those with abrupt exogenous shocks may render the estimation of f_t unstable. Moreover, the current implementation uses aggregate summary statistics, which may not fully capture heterogeneity across individual units. Future work is suggested in three areas: (a) developing robust state‑evolution estimators for short or noisy time series, (b) hybrid methods that combine partial network information with temporal dynamics, and (c) extending the validation to other domains such as public policy interventions, healthcare, and online platform experiments.

In summary, the study empirically validates that causal message passing can recover total treatment effects as accurately as network‑aware bipartite graph methods, even when no network data are available. This finding has important implications for experimental design and analysis in settings where collecting or sharing network information is impractical, offering a cost‑effective pathway to unbiased inference under interference.