Dynamic factor analysis for sparse and irregular longitudinal data: an application to metabolite measurements in a COVID-19 study

It is of scientific interest to identify essential biomarkers in biological processes underlying diseases to facilitate precision medicine. Factor analysis (FA) has long been used to address this goal: by assuming latent biological pathways drive the activity of measurable biomarkers, a biomarker is more influential if its absolute factor loading is larger. Although correlation between biomarkers has been properly handled under this framework, correlation between latent pathways are often overlooked, as one classical assumption in FA is the independence between factors. However, this assumption may not be realistic in the context of pathways, as existing biological knowledge suggests that pathways interact with one another rather than functioning independently. Motivated by sparsely and irregularly collected longitudinal measurements of metabolites in a COVID-19 study of large sample size, we propose a dynamic factor analysis model that can account for the potential cross-correlations between pathways, through a multi-output Gaussian processes (MOGP) prior on the factor trajectories. To mitigate against overfitting caused by sparsity of longitudinal measurements, we introduce a roughness penalty upon MOGP hyperparameters and allow for non-zero mean functions. To estimate these hyperparameters, we develop a stochastic expectation maximization (StEM) algorithm that scales well to the large sample size. In our simulation studies, StEM leads across all sample sizes considered to a more accurate and stable estimate of the MOGP hyperparameters than a comparator algorithm used in previous research. Application to the motivating example identifies a kynurenine pathway that affects the clinical severity of patients with COVID-19. In particular, a novel biomarker taurine is discovered, which has been receiving increased attention clinically, yet its role was overlooked in a previous analysis.

💡 Research Summary

The paper addresses the challenge of identifying key biomarkers from sparsely and irregularly sampled longitudinal metabolite data collected from COVID‑19 patients. Traditional factor analysis (FA) assumes independence among latent factors (biological pathways), which is biologically implausible because pathways often interact. To overcome this limitation, the authors propose a dynamic factor analysis model that incorporates cross‑correlations among factors by placing a multi‑output Gaussian process (MOGP) prior on the latent factor trajectories. The MOGP is constructed via a convolution process framework, using shared and output‑specific white‑noise base processes convolved with Gaussian kernels. This structure yields a flexible cross‑covariance function while ensuring positive‑definiteness.

Because sparse measurements can lead to over‑fitting of Gaussian processes, the model penalizes the smoothness hyperparameters (variance and length‑scale parameters) and allows non‑zero mean functions for each factor, thereby regularizing the trajectories. For the factor loadings, a Bayesian sparse factor analysis (BSFA) is employed: each loading is expressed as the product of a binary inclusion indicator and a continuous coefficient. A Bernoulli‑Beta prior on the indicator and a Normal‑Inverse‑Gamma prior on the coefficient promote exact sparsity, eliminating the need for arbitrary thresholding. Subject‑specific biomarker means and residual variances also receive conjugate priors.

Parameter estimation is performed with a stochastic expectation‑maximization (StEM) algorithm. In the E‑step, latent factor trajectories are sampled via a Gibbs/MCMC scheme that naturally accommodates irregular observation times. In the M‑step, the MOGP hyperparameters are updated using the sampled trajectories, yielding an algorithm that scales well to the study’s size (n≈101 subjects, p=35 metabolites). Simulation experiments across a range of sample sizes demonstrate that StEM provides more accurate and stable hyperparameter estimates than the Monte‑Carlo EM (MCEM) method previously used, with an average speed‑up of about 20×.

Applying the method to the COVID‑19 cohort, the authors recover the known kynurenine pathway associated with disease severity and, importantly, identify taurine as a novel biomarker linked to this pathway. Taurine has attracted clinical interest but was missed in earlier univariate analyses. The findings illustrate the advantage of modeling correlated pathways and sparsity jointly. An R package, DFA4SIL, implements the full methodology, facilitating broader adoption. Overall, the work delivers a statistically rigorous, computationally efficient framework for high‑dimensional, sparsely observed longitudinal data, advancing precision medicine efforts in infectious disease research.