On the Hardness of Conditional Independence Testing In Practice

Tests of conditional independence (CI) underpin a number of important problems in machine learning and statistics, from causal discovery to evaluation of predictor fairness and out-of-distribution robustness. Shah and Peters (2020) showed that, contrary to the unconditional case, no universally finite-sample valid test can ever achieve nontrivial power. While informative, this result (based on “hiding” dependence) does not seem to explain the frequent practical failures observed with popular CI tests. We investigate the Kernel-based Conditional Independence (KCI) test - of which we show the Generalized Covariance Measure underlying many recent tests is nearly a special case - and identify the major factors underlying its practical behavior. We highlight the key role of errors in the conditional mean embedding estimate for the Type-I error, while pointing out the importance of selecting an appropriate conditioning kernel (not recognized in previous work) as being necessary for good test power but also tending to inflate Type-I error.

💡 Research Summary

This paper tackles the practical difficulties of conditional independence (CI) testing, a cornerstone of many modern machine learning and statistical tasks such as causal discovery, fairness evaluation, and out‑of‑distribution robustness. While Shah and Peters (2020) proved a striking impossibility result—no universally finite‑sample valid CI test can achieve non‑trivial power—the authors argue that this theoretical construction, which “hides” dependence in a very specific way, does not explain the systematic failures observed with popular CI tests in real‑world settings.

The authors focus on the Kernel‑based Conditional Independence (KCI) test, originally introduced by Zhang et al. (2011), and on the Generalized Covariance Measure (GCM) family, which includes weighted variants and has been the basis for many recent CI procedures. They first restate conditional independence using Daudin’s functional definition: for all square‑integrable functions f(A,C) and g(B,C) and any weighting function w(C), the conditional covariance E