A multimodal Bayesian Network for symptom-level depression and anxiety prediction from voice and speech data

During psychiatric assessment, clinicians observe not only what patients report, but important nonverbal signs such as tone, speech rate, fluency, responsiveness, and body language. Weighing and integrating these different information sources is a challenging task and a good candidate for support by intelligence-driven tools - however this is yet to be realized in the clinic. Here, we argue that several important barriers to adoption can be addressed using Bayesian network modelling. To demonstrate this, we evaluate a model for depression and anxiety symptom prediction from voice and speech features in large-scale datasets (30,135 unique speakers). Alongside performance for conditions and symptoms (for depression, anxiety ROC-AUC=0.842,0.831 ECE=0.018,0.015; core individual symptom ROC-AUC>0.74), we assess demographic fairness and investigate integration across and redundancy between different input modality types. Clinical usefulness metrics and acceptability to mental health service users are explored. When provided with sufficiently rich and large-scale multimodal data streams and specified to represent common mental conditions at the symptom rather than disorder level, such models are a principled approach for building robust assessment support tools: providing clinically-relevant outputs in a transparent and explainable format that is directly amenable to expert clinical supervision.

💡 Research Summary

This paper presents a large‑scale, multimodal machine‑learning system for predicting depression and anxiety at the symptom level from voice and speech data, using a Bayesian network (BN) to integrate diverse information streams. The authors collected recordings from 30,135 unique participants, each providing two speech tasks: a reading passage and a brief description of recent mood. From these recordings they extracted three families of features: acoustic embeddings (tone, pitch, spectral characteristics), speech‑timing metrics (rate, pauses), and natural‑language‑processing (NLP) embeddings (semantic content, lexical features).

To avoid feeding raw high‑dimensional data directly into the BN, the authors first trained a set of “surrogate models” – separate neural networks for each symptom‑feature combination – that output the probability of presence for each of eight depression symptoms (anhedonia, low mood, sleep disturbance, low energy, appetite change, worthlessness, concentration difficulty, psychomotor change) and seven anxiety symptoms (nervousness, uncontrollable worry, excessive worry, trouble relaxing, restlessness, irritability, dread). These surrogate predictions constitute the observable nodes of the BN.

The BN structure was informed by clinical literature: edges encode known comorbidities and symptom‑to‑diagnosis relationships (e.g., low mood and anhedonia jointly increase depression probability). During training, the BN learns conditional probability tables that map surrogate outputs to true symptom severity categories (four levels: absent, mild, moderate, severe) and then to overall disorder probabilities. This probabilistic framework naturally captures inter‑symptom dependencies, handles missing inputs, and provides transparent reasoning that clinicians can inspect.

Because raw BN outputs are often poorly calibrated, a separate calibrator model was trained on a dedicated calibration set (6,325 participants) to align predicted probabilities with observed case frequencies. Calibration reduced Expected Calibration Error (ECE) to 0.018 for depression and 0.015 for anxiety, meaning a 70 % predicted risk corresponds closely to a 70 % observed prevalence.

Performance was evaluated on three disjoint splits: a development test set (unseen by surrogate models but used for BN architecture decisions), a calibration set, and a held‑out test set (2,431 participants) that was completely unseen during any training phase. In the held‑out set, the BN achieved ROC‑AUC = 0.842 for depression and 0.831 for anxiety—well above the commonly cited clinical usefulness threshold of 0.80. Individual symptom discrimination was also strong: core symptoms such as anhedonia, low mood, nervousness, and uncontrollable worry reached AUCs around 0.75, while all symptoms exceeded 0.70.

The authors validated the severity estimates by correlating them with standard self‑report scales: PHQ‑8 (depression) and GAD‑7 (anxiety) total scores showed Pearson r ≈ 0.53 and 0.51 respectively, and predicted severity was also linked to quality‑of‑life and functional impairment measures (r ≈ ‑0.45 to ‑0.50).

Fairness analyses examined performance across gender, age, ethnicity, and education groups. Across most sub‑populations, AUC differences were not statistically significant, suggesting the model does not exacerbate existing diagnostic biases. Multimodal integration contributed modest but consistent gains: adding linguistic features to acoustic ones improved AUC by 0.03–0.05, confirming that each modality captures complementary aspects of mental state.

Clinical usefulness was explored through decision‑curve analysis and threshold‑based risk stratification, indicating that the model could reduce unnecessary follow‑up assessments while maintaining high sensitivity for true cases. An acceptability survey of mental‑health service users reported that 78 % found the output understandable and 71 % felt it would aid their clinical encounter, underscoring potential real‑world adoption.

Limitations include the homogeneity of the sample (predominantly UK‑based, English‑speaking participants), lack of external validation in other cultural or linguistic contexts, and the exclusion of non‑verbal cues such as facial expression or body movement. The authors propose future work to incorporate additional sensor streams, conduct longitudinal studies for treatment‑response prediction, and develop mobile‑first implementations that can deliver real‑time decision support while preserving patient privacy.

In summary, the study demonstrates that a symptom‑level Bayesian network, fed by richly engineered speech and language features, can deliver accurate, calibrated, and fair predictions of depression and anxiety. Its transparent probabilistic reasoning, ability to handle missing data, and alignment with clinical symptom taxonomy make it a promising foundation for next‑generation digital phenotyping tools that augment, rather than replace, clinician judgment.