Causal-Driven Feature Evaluation for Cross-Domain Image Classification

Out-of-distribution (OOD) generalization remains a fundamental challenge in real-world classification, where test distributions often differ substantially from training data. Most existing approaches pursue domain-invariant representations, implicitly assuming that invariance implies reliability. However, features that are invariant across domains are not necessarily causally effective for prediction. In this work, we revisit OOD classification from a causal perspective and propose to evaluate learned representations based on their necessity and sufficiency under distribution shift. We introduce an explicit segment-level framework that directly measures causal effectiveness across domains, providing a more faithful criterion than invariance alone. Experiments on multi-domain benchmarks demonstrate consistent improvements in OOD performance, particularly under challenging domain shifts, highlighting the value of causal evaluation for robust generalization.

💡 Research Summary

The paper tackles the long‑standing problem of out‑of‑distribution (OOD) generalization in image classification from a causal perspective. While most domain‑generalization methods focus on learning representations that are invariant across training domains, the authors argue that invariance alone does not guarantee that a feature is causally useful for prediction. Spurious features can be stable across domains yet irrelevant—or even harmful—when the test distribution shifts. To address this gap, the authors introduce a two‑stage framework that (1) learns a structured latent space composed of K semantically meaningful segments and (2) evaluates each segment’s causal contribution using the probability of necessity and sufficiency (PNS), a concept originally proposed by Pearl.

In the first stage, an encoder E maps an input image x to a latent code z ∈ ℝ^{K×d}, which is partitioned into K blocks z^{(1)},…,z^{(K)}. A decoder G reconstructs the image from z, and a cycle‑consistency loss forces the same block to represent the same abstract factor across all training domains. Formally, for each block k there exists an abstract factor C(k) and domain‑specific invertible maps ψ^{(k)}_D such that z^{(k)} = ψ^{(k)}_D(C(k)). This semantic alignment guarantees that a “do‑intervention” on block k corresponds to an intervention on the same underlying causal variable regardless of the domain.

The second stage fixes a classifier g that operates on the full latent code and measures the effect of intervening on each block. For block k, the authors replace it with a random value \tilde{z}^{(k)} (or set it to zero) while keeping the other blocks unchanged, then observe whether the predicted label changes. Necessity (N) is defined as the probability that removing the block changes the prediction; sufficiency (S) is the probability that the block alone yields the correct prediction. The PNS score is N × S. Because the blocks are semantically aligned, the PNS estimated on the source domains is expected to transfer to unseen target domains.

The paper also provides a rigorous discussion of why naïve coordinate‑wise interventions on unconstrained representations are ill‑defined: arbitrary reparameterizations can mix semantics across coordinates, breaking the causal meaning of a “do‑operation.” By contrast, segment‑level interventions on a shared latent chart are invariant to such reparameterizations, making the causal scores well‑posed.

Empirical evaluation is performed on four standard multi‑domain image benchmarks: PACS, VLCS, Office‑Home, and DomainNet. The authors compare against strong baselines such as Invariant Risk Minimization (IRM), Group Distributionally Robust Optimization (GroupDRO), SWAD, and CORAL. Results show consistent improvements in average test‑domain accuracy, with gains ranging from 2 to 5 percentage points. The advantage is most pronounced under severe domain shifts (e.g., Sketch → Photo), where spurious invariant features tend to dominate. An ablation study correlates individual segment PNS scores with OOD performance, confirming that high‑PNS segments are indeed the ones that drive the observed accuracy gains.

In summary, the work makes three key contributions: (1) it formalizes a causal evaluation metric (PNS) for representation segments, (2) it proposes a practical way to obtain semantically aligned latent segments via a generative factorization, and (3) it demonstrates that selecting or weighting features by their causal effectiveness yields superior OOD generalization. The paper opens several avenues for future research, including automated discovery of aligned segments, extension to non‑image modalities, and integration of causal evaluation into end‑to‑end training objectives.