Assessing the Distributional Fidelity of Synthetic Chest X-rays using the Embedded Characteristic Score

Chest X-ray (CXR) images are among the most commonly used diagnostic imaging modalities in clinical practice. Stringent privacy constraints often limit the public dissemination of patient CXR images, contributing to the increasing use of synthetic images produced by deep generative models for data sharing and training machine learning models. Given the high-stakes downstream applications of CXR images, it is crucial to evaluate how faithfully synthetic images reflect the underlying target distribution. We propose the embedded characteristic score (ECS), a flexible evaluation procedure that compares synthetic and patient CXR samples through characteristic function transforms of feature embeddings. The choice of embedding can be tailored to the clinical or scientific context of interest. By leveraging the behavior of characteristic functions near the origin, ECS is sensitive to differences in higher moments and distribution tails, aspects that are often overlooked by commonly used evaluation metrics such as the Fréchet Inception Distance (FID). We establish theoretical properties of ECS and describe a calibration strategy based on a simple resampling procedure. We compare the empirical performance of ECS against FID via simulations and standard benchmark imaging datasets. Assessing synthetic CXR images with ECS uncovers clinically relevant distributional discrepancies relative to patient CXR images. These results highlight the importance of reliable evaluation of synthetic data that inform high-stakes decisions.

💡 Research Summary

The paper addresses a critical gap in the evaluation of synthetic medical images, specifically chest X‑ray (CXR) scans, by introducing the Embedded Characteristic Score (ECS). While synthetic images are increasingly used to overcome privacy and regulatory constraints, existing quality metrics such as the Fréchet Inception Distance (FID) focus only on first‑ and second‑order moments under a Gaussian assumption and therefore miss discrepancies in higher moments and tail behavior—features that often carry rare pathological information. Human visual Turing tests, although useful for assessing perceptual realism, are also limited in detecting distributional mismatches.

ECS is built on two ideas: (1) an embedding function f that maps each image to a p‑dimensional feature vector, which can be a deep network representation (e.g., Inception‑v3) or a clinically curated set of anatomical/pathological measurements; and (2) the characteristic function φ(t)=E