Metric Hub: A metric library and practical selection workflow for use-case-driven data quality assessment in medical AI

Machine learning (ML) in medicine has transitioned from research to concrete applications aimed at supporting several medical purposes like therapy selection, monitoring and treatment. Acceptance and effective adoption by clinicians and patients, as well as regulatory approval, require evidence of trustworthiness. A major factor for the development of trustworthy AI is the quantification of data quality for AI model training and testing. We have recently proposed the METRIC-framework for systematically evaluating the suitability (fit-for-purpose) of data for medical ML for a given task. Here, we operationalize this theoretical framework by introducing a collection of data quality metrics - the metric library - for practically measuring data quality dimensions. For each metric, we provide a metric card with the most important information, including definition, applicability, examples, pitfalls and recommendations, to support the understanding and implementation of these metrics. Furthermore, we discuss strategies and provide decision trees for choosing an appropriate set of data quality metrics from the metric library given specific use cases. We demonstrate the impact of our approach exemplarily on the PTB-XL ECG-dataset. This is a first step to enable fit-for-purpose evaluation of training and test data in practice as the base for establishing trustworthy AI in medicine.

💡 Research Summary

The paper addresses a critical gap in trustworthy medical artificial intelligence: the lack of systematic, quantitative methods for assessing the quality of data used to train and test machine‑learning (ML) models. Building on the previously introduced METRIC‑framework, which defines 26 data‑quality dimensions grouped into five clusters (measurement process, timeliness, representativeness, informativeness, and consistency), the authors operationalize the framework by creating a practical “Metric Hub.”

First, they performed an extensive literature review and a multi‑institution focus‑group process to collect 60 quantitative metrics that can be applied to 14 of the METRIC dimensions that are amenable to numerical evaluation. Each metric is documented in a standardized “metric card” that includes a concise definition, value range, applicability conditions, required prerequisites, common pitfalls, and implementation recommendations. The cards are hosted on an online platform (Metric Hub) to facilitate reuse and future extension.

The metrics are organized into seven logical groups: (1) measurement‑process metrics (e.g., accuracy, repeatability, limits of detection/quantification), (2) consistency metrics (syntactic consistency, homogeneity), (3) representativeness metrics (dataset size, granularity, class balance, variety), (4) timeliness metrics (currency, effective sample size), (5) informativeness metrics (feature importance, informative missingness), (6) distribution metrics (statistical tests and distance measures such as Kolmogorov‑Smirnov, Jensen‑Shannon, Maximum Mean Discrepancy), and (7) correlation‑coefficient metrics (Cohen’s κ, Pearson, Spearman, etc.). Some metrics belong to multiple dimensions, reflecting the overlapping nature of data‑quality concepts.

Recognizing that computing every metric for a given project is inefficient and often infeasible, the authors introduce decision‑tree based workflows for metric selection. Each of the 14 quantitative dimensions has its own tree, which asks a series of binary or categorical questions about the use case: data modality (tabular, image, time‑series, multimodal, text), variable type (continuous, categorical), ML task (classification, regression, segmentation), availability of repeated measurements or ground‑truth, presence of blank‑sample measurements, and so forth. The leaf nodes of the tree correspond to specific metric(s) that are appropriate given the answers. For example, the “Accuracy” tree guides the user to Bland‑Altman analysis when repeated measurements under the same conditions are available, or to limits of detection/quantification when only single measurements exist. This approach yields a defensible, reproducible metric set tailored to the specific clinical AI scenario while avoiding unnecessary computation.

To demonstrate feasibility, the authors applied the Metric Hub to the PTB‑XL electrocardiogram (ECG) dataset, a large public collection used for multiclass arrhythmia classification. They derived a use‑case‑specific metric set for the classification task and computed the metrics on the original dataset as well as on three synthetically perturbed subsets that altered sex balance, device distribution, and target‑class balance. The analysis revealed measurable differences across dimensions such as representativeness (class imbalance, device variety), consistency (distribution drift), and timeliness (currency). These differences correlated with changes in model performance, illustrating how quantitative data‑quality assessment can uncover hidden sources of bias or error that are not apparent from aggregate performance metrics alone.

The contributions of the work are threefold: (1) a curated, extensible library of 60 quantitative data‑quality metrics aligned with the METRIC‑framework, each described by a concise metric card; (2) a set of decision‑tree workflows that enable fit‑for‑purpose, model‑independent metric selection based on concrete use‑case attributes; and (3) a practical case study on a real‑world clinical dataset that validates the utility of the approach and highlights both its strengths and current limitations (e.g., dependence on availability of reference measurements).

By providing concrete tools for systematic, transparent, and regulatory‑compatible data‑quality evaluation, the Metric Hub moves the field toward a data‑centric paradigm of trustworthy medical AI, where the suitability of training and test data can be documented, monitored, and improved throughout the lifecycle of AI systems.