A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications

Deep learning has revolutionized biomedical research by providing sophisticated methods to handle complex, high-dimensional data. Multimodal deep learning (MDL) further enhances this capability by integrating diverse data types such as imaging, textual data, and genetic information, leading to more robust and accurate predictive models. In MDL, differently from early and late fusion methods, intermediate fusion stands out for its ability to effectively combine modality-specific features during the learning process. This systematic review aims to comprehensively analyze and formalize current intermediate fusion methods in biomedical applications. We investigate the techniques employed, the challenges faced, and potential future directions for advancing intermediate fusion methods. Additionally, we introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models. Through this review, we aim to provide a foundational framework for future research and practical applications in the dynamic field of MDL.

💡 Research Summary

This systematic review provides a comprehensive analysis of intermediate fusion techniques within multimodal deep learning (MDL) applied to biomedical problems. The authors begin by contrasting early, late, and intermediate fusion, formally defining intermediate fusion as the integration of modality‑specific intermediate representations (h₁,…,hₙ) via a fusion function F before the final prediction network f processes the combined representation. Using a rigorously defined inclusion/exclusion protocol—English, peer‑reviewed studies that employ deep learning‑based intermediate fusion on at least two biomedical modalities—the authors searched PubMed, IEEE Xplore, Scopus, and Google Scholar with a query that combined terms from three categories: “Multimodal Deep Learning,” “Biomedical,” and “Intermediate Fusion.” This yielded roughly 150 original research articles up to 2024, whose detailed characteristics are provided in supplementary tables.

The review categorizes current intermediate fusion methods into four principal families: (1) simple concatenation of intermediate feature vectors, (2) attention‑based mechanisms that learn modality‑wise importance weights, (3) graph or message‑passing networks that explicitly model inter‑modality relationships, and (4) probabilistic generative approaches (e.g., variational autoencoders, GANs) that can handle label scarcity and quantify uncertainty. Comparative analysis shows that attention‑based models often outperform simple concatenation when data are noisy or imbalanced, while graph‑based methods excel in capturing complex biological interactions such as gene‑expression‑image correlations.

Despite these advances, the authors identify several persistent challenges. First, the black‑box nature of deep intermediate fusion hampers clinical interpretability; second, large labeled datasets are frequently required, conflicting with privacy constraints and the rarity of certain medical conditions; third, balancing contributions from heterogeneous modalities remains an open optimization problem, leading to dominance of a single modality in many implementations; fourth, computational and memory demands are high, limiting deployment on edge devices or in real‑time clinical workflows.

To address these gaps, the paper proposes five future research directions: (a) dynamic modality weighting via meta‑learning or reinforcement‑learning strategies, (b) integration of explainable AI techniques (e.g., saliency maps, concept activation vectors) to improve transparency, (c) federated or privacy‑preserving learning frameworks that enable collaborative model training without sharing raw data, (d) model compression, quantization, and hardware‑aware architecture design for efficient inference, and (e) multi‑task and multi‑domain transfer learning to enhance generalization across disparate biomedical datasets.

A notable contribution is the introduction of a formal mathematical notation for intermediate fusion, which the authors argue can be readily extended beyond biomedical applications to any multimodal domain (e.g., vision‑language, audio‑text). By providing a clear taxonomy, a reproducible notation, and a curated dataset of representative studies, this review fills a critical gap in the literature, offering researchers a solid foundation for developing more accurate, interpretable, and scalable multimodal deep learning systems in healthcare and beyond.