Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning

Multimodal graphs are gaining increasing attention due to their rich representational power and wide applicability, yet they introduce substantial challenges arising from severe modality confusion. To address this issue, we propose NSG (Node Splitting Graph)-MoE, a multimodal graph learning framework that integrates a node-splitting and graph-rewiring mechanism with a structured Mixture-of-Experts (MoE) architecture. It explicitly decomposes each node into modality-specific components and assigns relation-aware experts to process heterogeneous message flows, thereby preserving structural information and multimodal semantics while mitigating the undesirable mixing effects commonly observed in general-purpose GNNs. Extensive experiments on three multimodal benchmarks demonstrate that NSG-MoE consistently surpasses strong baselines. Despite incorporating MoE – which is typically computationally heavy – our method achieves competitive training efficiency. Beyond empirical results, we provide a spectral analysis revealing that NSG performs adaptive filtering over modality-specific subspaces, thus explaining its disentangling behavior. Furthermore, an information-theoretic analysis shows that the architectural constraints imposed by NSG reduces mutual information between data and parameters and improving generalization capability.

💡 Research Summary

The paper tackles the pervasive problem of modality confusion in multimodal graph learning by introducing a novel framework called NSG‑MoE (Node Splitting Graph – Mixture of Experts). Traditional multimodal graph approaches concatenate all modality embeddings into a single node feature vector before feeding them into a graph neural network (GNN). This early‑fusion strategy ignores the fact that different modalities (e.g., text, image, audio) live in distinct embedding spaces and have heterogeneous noise characteristics, leading to the blurring of modality‑specific signals during message passing.

NSG‑MoE first transforms a homogeneous multimodal graph G = (V, E, M) into a heterogeneous graph G* by splitting each original node v into |M| sub‑nodes ⟨v, i⟩, each representing one modality. Modality‑specific feature slices are linearly projected into a shared latent dimension d, ensuring comparable representations while preserving each modality’s intrinsic distribution.

The graph is then rewired in two ways. Intra‑node edges connect all sub‑nodes belonging to the same original node, forming an m‑clique that enables information exchange among modalities of the same entity. Inter‑node edges are created for every original edge (u, v): every sub‑node of u is linked to every sub‑node of v. These inter‑node edges are further divided into self‑type (same modality) and cross‑type (different modalities) connections, effectively modeling both homogeneous and heterogeneous relational patterns across the graph.

On top of this restructured graph, the authors embed a Graph Mixture‑of‑Experts (Graph‑MoE) module. Each expert is implemented as a heterogeneous GNN (HGNN) and specializes in a particular relational pattern (e.g., cross‑modal interactions, same‑modality long‑range dependencies). A gating network dynamically routes each sub‑node’s message to the most appropriate experts, achieving sparse activation and keeping computational overhead modest despite the use of MoE.

Theoretical contributions include a spectral analysis that reduces the heterogeneous GNN operations to a linear form and shows that node splitting induces independent low‑pass filters on each modality subspace. The filter strength is adaptively controlled by the gating mechanism, resulting in modality‑aware smoothing that suppresses noise while preserving signal. An information‑theoretic analysis demonstrates that the architectural constraints reduce the mutual information between data and parameters, yielding tighter generalization bounds.

Empirical evaluation spans three multimodal benchmarks: a web‑page graph with text and images, a medical graph combining imaging and lab measurements, and a public multimodal knowledge graph. The model is tested on node classification and link prediction tasks. NSG‑MoE consistently outperforms strong baselines by 3.2 %–5.8 % absolute accuracy, while training time remains comparable (≈1.1× that of standard GNNs) thanks to the sparsity of the MoE. Ablation studies confirm the importance of intra‑node cliques, cross‑type edges, and the MoE component.

In summary, the work reframes modalities as a form of heterogeneity, resolves modality confusion through explicit node splitting and graph rewiring, and augments the system with relation‑aware experts. This combination yields a flexible, scalable, and theoretically grounded solution for multimodal graph learning, with potential impact across domains such as multimodal knowledge graphs, biomedical networks, and social media analysis.