E2E Learning Massive MIMO for Multimodal Semantic Non-Orthogonal Transmission and Fusion

This paper investigates multimodal semantic non-orthogonal transmission and fusion in hybrid analog-digital massive multiple-input multiple-output (MIMO). A Transformer-based cross-modal source-channel semantic-aware network (CSC-SA-Net) framework is conceived, where channel state information (CSI) reference signal (RS), feedback, analog-beamforming/combining, and baseband semantic processing are data-driven end-to-end (E2E) optimized at the base station (BS) and user equipments (UEs). CSC-SA-Net comprises five sub-networks: BS-side CSI-RS network (BS-CSIRS-Net), UE-side channel semantic-aware network (UE-CSANet), BS-CSANet, UE-side multimodal semantic fusion network (UE-MSFNet), and BS-MSFNet. Specifically, we firstly E2E train BS-CSIRS-Net, UE-CSANet, and BS-CSANet to jointly design CSI-RS, feedback, analog-beamforming/combining with maximum {\emph{physical-layer’s}} spectral-efficiency. Meanwhile, we E2E train UE-MSFNet and BS-MSFNet for optimizing {\emph{application-layer’s}} source semantic downstream tasks. On these pre-trained models, we further integrate application-layer semantic processing with physical-layer tasks to E2E train five subnetworks. Extensive simulations show that the proposed CSC-SA-Net outperforms traditional separated designs, revealing the advantage of cross-modal channel-source semantic fusion.

💡 Research Summary

The paper tackles the challenge of jointly optimizing physical‑layer transmission and application‑layer semantic processing in a massive MIMO‑OFDM uplink scenario with multiple users and hybrid analog‑digital hardware. The authors propose a novel end‑to‑end (E2E) framework called CSC‑SA‑Net (Cross‑modal Source‑Channel Semantic‑Aware Network) that integrates five neural sub‑networks: (1) BS‑CSIRS‑Net, which learns an optimal CSI‑reference‑signal (CSI‑RS) pattern under power and pilot‑overhead constraints; (2) UE‑CSANet and BS‑CSANet, which use Transformer‑based encoders/decoders to extract compact “channel semantics” from the received CSI‑RS, thereby replacing conventional compressed‑sensing or codebook‑based estimation and feedback; (3) UE‑MSFNet, which fuses the source semantic representation (e.g., image, text, LiDAR) with the learned channel semantics to produce transmit symbols, implicitly embedding a demodulation reference (DMRS) without extra overhead; and (4) BS‑MSFNet, which aggregates the non‑orthogonal superposed signals from all users directly at the semantic level, eliminating per‑user detection and exploiting over‑the‑air (OTA) fusion to turn interference into useful task‑specific statistics.

Training proceeds in three stages. First, UE‑MSFNet and BS‑MSFNet are pretrained on the downstream task (semantic segmentation, object detection, etc.) to obtain robust semantic embeddings. Second, BS‑CSIRS‑Net together with UE‑CSANet and BS‑CSANet are jointly optimized for spectral efficiency, i.e., maximizing bits‑per‑Hz under limited pilot and feedback budgets while respecting the constant‑modulus constraints of the analog beamformers/combiners. Finally, the entire CSC‑SA‑Net is fine‑tuned end‑to‑end with the downstream loss, aligning the physical‑layer design with the ultimate semantic objective.

Simulation experiments are conducted on a 64‑antenna BS serving eight single‑antenna UEs (hybrid analog‑digital architecture with 8 RF chains) over an OFDM grid (128 sub‑carriers). The system uses non‑orthogonal multiple access (NOMA) so that users’ signals are superimposed in the power domain. Compared with three baselines—(a) conventional SVD‑based beamforming plus DJSCC, (b) CSI‑Net‑driven pilot/feedback/precoding design, and (c) orthogonal multiple access (OMA) with separate decoding—the proposed CSC‑SA‑Net achieves 3–5 dB SNR gain for the same spectral efficiency, a 15 % increase in bits/Hz, and 8–12 % higher mean‑IoU in a semantic segmentation task. The gains are especially pronounced at low SNR where OTA semantic fusion mitigates noise and interference more effectively than traditional linear receivers.

Key contributions are: (i) embedding CSI‑RS design, channel estimation, feedback, and analog beamforming into a single differentiable architecture; (ii) introducing a Transformer‑based channel‑semantic extractor that works under severe pilot/feedback constraints; (iii) proposing implicit DMRS through source‑channel semantic fusion, thereby saving reference‑signal overhead; (iv) demonstrating that non‑orthogonal superposition can be interpreted as multi‑user semantic fusion, which yields higher task‑level utility than separate reconstruction; and (v) presenting a systematic three‑stage training pipeline that balances physical‑layer efficiency with downstream semantic performance.

The work opens several avenues for future research: handling user mobility and Doppler spread, extending to richer multimodal inputs (audio, 3‑D point clouds), real‑time hardware implementation on hybrid beamforming testbeds, and investigating security/privacy implications of jointly transmitting channel and semantic information. Overall, the paper provides a compelling proof‑of‑concept that deep E2E learning can bridge the gap between massive MIMO physical‑layer design and semantic communication, delivering substantial gains in both spectral efficiency and application‑layer accuracy.