Beyond Pooling: Matching for Robust Generalization under Data Heterogeneity

Pooling heterogeneous datasets across domains is a common strategy in representation learning, but naive pooling can amplify distributional asymmetries and yield biased estimators, especially in settings where zero-shot generalization is required. We propose a matching framework that selects samples relative to an adaptive centroid and iteratively refines the representation distribution. The double robustness and the propensity score matching for the inclusion of data domains make matching more robust than naive pooling and uniform subsampling by filtering out the confounding domains (the main cause of heterogeneity). Theoretical and empirical analyses show that, unlike naive pooling or uniform subsampling, matching achieves better results under asymmetric meta-distributions, which are also extended to non-Gaussian and multimodal real-world settings. Most importantly, we show that these improvements translate to zero-shot medical anomaly detection, one of the extreme forms of data heterogeneity and asymmetry. The code is available on https://github.com/AyushRoy2001/Beyond-Pooling.

💡 Research Summary

The paper addresses a fundamental problem in multi‑domain learning: naïve pooling of heterogeneous datasets often amplifies distributional asymmetries and leads to biased estimators, especially when zero‑shot generalization is required. To mitigate this, the authors propose a matching framework that iteratively selects entire domains based on their proximity to an adaptive centroid and then updates that centroid with the selected samples.

Problem Setting

• Each domain (k) is generated from a Gaussian (Q_k = \mathcal N(\mu_k,\sigma^2 I_d)).
• The domain means (\mu_k) are i.i.d. draws from a meta‑distribution (D_\mu) with mean (\mu^\ast) and covariance (\Sigma_\mu).
• The target test distribution is isotropic Gaussian (\mathcal N(\mu^\ast,\sigma^2 I_d)).

Three pooling strategies are defined: (1) naïve pooling (all samples), (2) uniform subsampling (randomly pick domains and a fixed number of samples per domain), and (3) matching (include a domain only if (|\mu_k - c_t| < \tau), where (c_t) is the current centroid).

Theoretical Contributions

1. Asymptotic Regime (K → ∞) – Theorem 1 shows that all three strategies converge to a distribution centered at (\mu^\ast). However, naïve pooling and subsampling retain the inter‑domain variance (\Sigma_\mu) and converge to (\mathcal N(\mu^\ast,\sigma^2 I_d + \Sigma_\mu)). Matching, by contrast, filters out (\Sigma_\mu) and converges to the exact target (\mathcal N(\mu^\ast,\sigma^2 I_d)). This demonstrates that matching uniquely eliminates domain‑level heterogeneity.

2. Finite‑K Symmetric Meta‑Distribution – Under a symmetric (D_\mu) (Definition 4), Theorem 2 proves that all three methods are unbiased estimators of (\mu^\ast) for any finite number of domains. Thus, when domain means are balanced around the target, naïve pooling does not suffer bias.

3. Finite‑K Asymmetric Meta‑Distribution – When (D_\mu) is asymmetric, naïve pooling and subsampling can incur systematic bias because they do not condition domain inclusion on the target. Matching satisfies the causal inference assumptions of ignorability, positivity, and consistency by using a propensity‑score‑like inclusion rule (W_k = \mathbf 1{|\mu_k - c_t| < \tau}). This yields double robustness: consistency is guaranteed if either the propensity model (the inclusion rule) or the outcome model (centroid update) is correctly specified. Consequently, mild misspecification of (\tau) does not break the estimator, making matching stable under realistic heterogeneity.

4. Exchangeability – All strategies are symmetric functions of the i.i.d. draws ({\mu_k}), but only matching operationalizes exchangeability to achieve conditional balance with respect to (\mu^\ast).

Algorithmic Details

• Initialize centroid (c_0 = \mu^\ast) (or an estimate).
• At iteration (t), compute inclusion indicator (W_k^{(t)} = \mathbf 1{|\mu_k - c_t| < \tau}).
• Accept all samples from domains with (W_k^{(t)} = 1).
• Update centroid (c_{t+1}) as the mean of all accepted samples.
• Repeat until convergence (centroid stabilizes).

Empirical Validation

• Synthetic Experiments: Varying the asymmetry of (D_\mu) and the number of domains, matching consistently yields lower estimation error and faster convergence than pooling or subsampling.
• Zero‑Shot Medical Anomaly Detection: Benchmarked on several heterogeneous medical imaging datasets (Chest‑XRay, Brain MRI, OCT, Liver CT). The authors evaluate three metrics: Domain Alignment (DA), Anomaly Classification (AC), and Anomaly Segmentation (AS). Matching achieves DA scores ≥ 4.0 across all tasks, outperforming MVFA, AnomalyCLIP, and BiLORA, which hover around 2–3. The AUC for anomaly detection also improves markedly, confirming that the method generalizes to real‑world, highly imbalanced, and multimodal data.

Practical Implications

• Matching provides a principled way to incorporate newly arriving domains without re‑training from scratch, which is valuable for federated or continual learning settings.
• The double‑robustness property ensures that even if the threshold (\tau) is not perfectly tuned, the estimator remains consistent, reducing the need for extensive hyper‑parameter search.
• By explicitly filtering out confounding domains, the approach mitigates fairness concerns that arise when certain institutions dominate the pooled data.

Conclusion
The paper convincingly demonstrates that naïve pooling is not a universally safe strategy for heterogeneous data. An adaptive, centroid‑based matching framework offers theoretical guarantees (asymptotic removal of inter‑domain variance, unbiasedness under symmetry, double robustness under asymmetry) and empirical superiority, especially in zero‑shot medical anomaly detection where data heterogeneity is extreme. The work bridges causal inference concepts (propensity‑score matching) with representation learning, opening a new avenue for robust multi‑domain model building.