Position: Evaluation of ECG Representations Must Be Fixed

This position paper argues that current benchmarking practice in 12-lead ECG representation learning must be fixed to ensure progress is reliable and aligned with clinically meaningful objectives. The field has largely converged on three public multi-label benchmarks (PTB-XL, CPSC2018, CSN) dominated by arrhythmia and waveform-morphology labels, even though the ECG is known to encode substantially broader clinical information. We argue that downstream evaluation should expand to include an assessment of structural heart disease and patient-level forecasting, in addition to other evolving ECG-related endpoints, as relevant clinical targets. Next, we outline evaluation best practices for multi-label, imbalanced settings, and show that when they are applied, the literature’s current conclusion about which representations perform best is altered. Furthermore, we demonstrate the surprising result that a randomly initialized encoder with linear evaluation matches state-of-the-art pre-training on many tasks. This motivates the use of a random encoder as a reasonable baseline model. We substantiate our observations with an empirical evaluation of three representative ECG pre-training approaches across six evaluation settings: the three standard benchmarks, a structural disease dataset, hemodynamic inference, and patient forecasting.

💡 Research Summary

The paper presents a critical examination of the current benchmarking practices in 12‑lead electrocardiogram (ECG) representation learning and proposes a comprehensive overhaul to align evaluation with clinically meaningful objectives. At present, the field relies almost exclusively on three public multi‑label datasets—PTB‑XL, CPSC2018, and CSN—which focus narrowly on arrhythmia detection and waveform morphology. While these benchmarks have facilitated rapid methodological progress, they ignore the broader spectrum of information encoded in ECG signals, such as structural heart disease, hemodynamic status, and patient‑level prognostic risk.

The authors first argue that limiting downstream evaluation to the three traditional benchmarks creates a feedback loop that favors methods optimized for a narrow set of labels, potentially obscuring the true utility of learned representations. They highlight two methodological shortcomings: (1) the pervasive use of macro‑AUROC as the sole headline metric, which aggregates per‑label performance into an unweighted mean and therefore masks clinically relevant variations, especially for rare labels; and (2) the near‑absence of uncertainty quantification, making it impossible to assess whether observed differences are statistically meaningful.

To remedy these issues, the paper proposes a set of best‑practice guidelines: (i) report per‑label AUROC, PR‑AUC, precision, and recall; (ii) accompany each metric with confidence intervals derived from bootstrapping or Bayesian methods; (iii) evaluate performance across multiple data‑size regimes (e.g., 1 %, 10 %, 100 % of the training set) to assess robustness; and (iv) always include a randomly initialized encoder with linear probing as a baseline.

Beyond methodological recommendations, the authors introduce a broader taxonomy of downstream tasks that better reflects the physiological role of the heart as an electromechanical pump. The taxonomy comprises four families: (1) traditional arrhythmia and waveform abnormalities; (2) structural disease detection (e.g., left‑ventricular ejection fraction, valvular disease) using datasets such as EchoNext; (3) hemodynamic state inference (e.g., pulmonary capillary wedge pressure, cardiac output) with data drawn from MIMIC‑IV or proprietary catheterization records; and (4) patient‑level forecasting, split into contemporaneous diagnosis (e.g., current low ejection fraction) and future risk prediction (e.g., development of heart failure within one year).

The empirical core of the paper evaluates three representative ECG pre‑training approaches—CLOCS (a contrastive method exploiting temporal and lead structure), MERL (multimodal alignment of ECG with clinical text), and D‑BETA (multimodal alignment plus reconstruction regularization)—across six evaluation settings: the three standard benchmarks, EchoNext (structural disease), a hemodynamic inference task, and a patient‑forecasting task. All methods are assessed via linear probing, and a randomly initialized encoder is included for comparison. Results show that when macro‑AUROC alone is considered, MERL and D‑BETA appear superior. However, once per‑label confidence intervals are taken into account, the performance differences largely disappear; in several cases CLOCS matches or exceeds the others, and the random encoder achieves comparable scores, especially when only 1 % of the training data is available.

These findings overturn the prevailing narrative that sophisticated self‑supervised pre‑training universally outperforms naïve baselines. Instead, the paper demonstrates that the choice of benchmark, evaluation metric, and reporting practice can dramatically alter method rankings. Consequently, the authors call for the community to adopt the proposed evaluation protocol, expand benchmark suites to include structural, hemodynamic, and prognostic tasks, and consistently benchmark against a random encoder.

In conclusion, the paper makes a compelling case that fixing evaluation—through richer clinical tasks, robust statistical reporting, and appropriate baselines—is essential for ECG representation learning to progress beyond academic competitions and deliver real clinical impact. By standardizing these practices, future research can produce reproducible, clinically relevant advances that truly leverage the wealth of information embedded in ECG signals.