Robust Real-Time Mortality Prediction in the Intensive Care Unit using Temporal Difference Learning

The task of predicting long-term patient outcomes using supervised machine learning is a challenging one, in part because of the high variance of each patient’s trajectory, which can result in the model over-fitting to the training data. Temporal difference (TD) learning, a common reinforcement learning technique, may reduce variance by generalising learning to the pattern of state transitions rather than terminal outcomes. However, in healthcare this method requires several strong assumptions about patient states, and there appears to be limited literature evaluating the performance of TD learning against traditional supervised learning methods for long-term health outcome prediction tasks. In this study, we define a framework for applying TD learning to real-time irregularly sampled time series data using a Semi-Markov Reward Process. We evaluate the model framework in predicting intensive care mortality and show that TD learning under this framework can result in improved model robustness compared to standard supervised learning methods. and that this robustness is maintained even when validated on external datasets. This approach may offer a more reliable method when learning to predict patient outcomes using high-variance irregular time series data.

💡 Research Summary

**
This paper introduces a novel framework that applies Temporal Difference (TD) learning—a reinforcement‑learning technique—to the problem of real‑time, long‑term mortality prediction in intensive care units (ICUs). Traditional supervised models often overfit because patient trajectories are highly variable; they map each observation directly to a fixed outcome label (e.g., 28‑day mortality) without accounting for the stochastic nature of state transitions. To address this, the authors formulate a Semi‑Markov Reward Process (Semi‑MRP) that allows state transitions to occur at irregular time intervals, reflecting the reality of clinical measurements. In the Semi‑MRP, the time gap (k) between successive “state markers” is drawn from an (unknown) distribution (D); the reward is defined only at the terminal state (1 for death, 0 for discharge), and the discount factor (\gamma) is set to 1, so the predicted risk is the average of all future risks.

Two large ICU cohorts are used: MIMIC‑IV (≈65 k patients, 84 k admissions) for training/validation/testing, and the Salzburg Intensive Care Database (SICdb, ≈21 k patients) for external validation. Each measurement is encoded as a 5‑tuple ({v, t, f, \Delta v, \Delta t}) (value, timestamp, feature, value change, time‑gap). Five separate multilayer perceptrons embed each component, the embeddings are summed, and the resulting sequence is processed by a CNN‑LSTM architecture. The CNN reduces sequence length, while the LSTM captures temporal dependencies; the final hidden state is decoded by two dense layers into a single mortality risk score (sigmoid output).

Six model families are trained on the same architecture: one TD model and five supervised baselines that predict mortality at fixed horizons (1, 3, 7, 14, 28 days). Supervised models use binary cross‑entropy with optional class‑balancing weights; a separate target network (as in DQN) stabilizes training. The TD model is trained using the loss derived from Equation (4), which bootstraps the value of the current state from the expected value of a future state sampled according to the chosen interval (k).

Performance is measured by AUROC on both internal and external test sets for each horizon. Internally, the TD‑24 hr model (state‑to‑state delay of 24 h) achieves the highest AUROC (≈0.88 for 28‑day mortality), outperforming supervised models (AUROC 0.81–0.84) and the clinical SOFA score (≈0.78). Crucially, on the external SICdb cohort the TD model’s AUROC drops only marginally (≈0.86), whereas supervised models suffer a larger degradation (AUROC 0.73–0.78), indicating superior robustness to dataset shift. One‑tailed paired Student’s t‑tests with Benjamini‑Yekutieli correction confirm that the TD model’s mean AUROC is significantly higher than each baseline (p < 0.05).

Key contributions: (1) a mathematically grounded Semi‑MRP formulation that accommodates irregularly sampled health data; (2) a direct empirical comparison between TD learning and conventional supervised learning using identical deep‑learning backbones; (3) demonstration that TD learning reduces variance and over‑fitting, yielding more stable performance across institutions.

Limitations include the binary reward design (death vs. discharge) which precludes modeling intermediate clinical events, the choice of (\gamma = 1) which treats all future risks equally (potentially unrealistic in practice), and the heuristic selection of the interval distribution (D) rather than learning it from data. Future work could explore multi‑step reward structures, time‑discounted risk weighting, and data‑driven estimation of (D) to further align the model with clinical decision‑making. Overall, the study provides compelling evidence that reinforcement‑learning‑based TD methods can enhance the reliability of mortality prediction models in high‑variance ICU settings.