MambaFusion: Adaptive State-Space Fusion for Multimodal 3D Object Detection

Reliable 3D object detection is fundamental to autonomous driving, and multimodal fusion algorithms using cameras and LiDAR remain a persistent challenge. Cameras provide dense visual cues but ill posed depth; LiDAR provides a precise 3D structure but sparse coverage. Existing BEV-based fusion frameworks have made good progress, but they have difficulties including inefficient context modeling, spatially invariant fusion, and reasoning under uncertainty. We introduce MambaFusion, a unified multi-modal detection framework that achieves efficient, adaptive, and physically grounded 3D perception. MambaFusion interleaves selective state-space models (SSMs) with windowed transformers to propagate the global context in linear time while preserving local geometric fidelity. A multi-modal token alignment (MTA) module and reliability-aware fusion gates dynamically re-weight camera-LiDAR features based on spatial confidence and calibration consistency. Finally, a structure-conditioned diffusion head integrates graph-based reasoning with uncertainty-aware denoising, enforcing physical plausibility, and calibrated confidence. MambaFusion establishes new state-of-the-art performance on nuScenes benchmarks while operating with linear-time complexity. The framework demonstrates that coupling SSM-based efficiency with reliability-driven fusion yields robust, temporally stable, and interpretable 3D perception for real-world autonomous driving systems.

💡 Research Summary

MambaFusion presents a novel end‑to‑end framework for multimodal 3D object detection that tightly integrates cameras and LiDAR in a bird’s‑eye‑view (BEV) space while addressing three long‑standing challenges: (1) inefficient global context modeling, (2) spatially invariant fusion that ignores sensor reliability, and (3) lack of physical reasoning and temporal stability.
The core of the system is a hybrid LiDAR encoder that alternates selective state‑space model (SSM) blocks—derived from the Mamba architecture—with windowed transformer layers. Raw point clouds are voxelized, serialized along a Hilbert curve to preserve locality, and processed at multiple scales. SSM blocks propagate long‑range dependencies in linear time, while windowed attention refines fine‑grained geometry, yielding a multi‑scale BEV feature map Q_BEV^L.
Parallel to this, multi‑view images are encoded by a shared backbone with an FPN. Deformable cross‑attention produces camera BEV tokens Q_BEV^C, and a temporal Mamba block aggregates these tokens across frames, again with linear complexity.
To mitigate real‑world calibration drift, a lightweight Multi‑Modal Token Alignment (MTA) module learns a residual offset ΔP between camera and LiDAR BEV tokens via cross‑attention, producing aligned tokens Q’_C and Q’_L.
Fusion is performed through bidirectional multi‑head attention followed by a spatial reliability gate. For each BEV cell, a descriptor combines point density, depth variance, occlusion score, multi‑view consistency, and ego‑distance. A small MLP maps this descriptor and the attended features to a sigmoid gate g(x,y). Additionally, each modality predicts a per‑cell log‑variance map; inverse‑variance weighting fuses the modalities, automatically down‑weighting high‑uncertainty regions. Early training freezes gradients on the variance maps for stability.
From the fused BEV map, modality‑specific heatmaps are linearly combined, and the top‑K peaks form initial proposals. A spatial graph connects each proposal to its k‑nearest neighbors; message passing enriches node features with structural descriptors (class‑size consistency, ground offset, LiDAR support, confidence). This graph reasoning suppresses implausible configurations such as overlapping or floating boxes.
Proposal confidence is further refined by a Structure‑Conditioned Diffusion (GCD) module. Each proposal’s feature–confidence pair undergoes a conditional denoising diffusion process where the noise level is modulated by a learned reliability score u_i derived from LiDAR support, ground proximity, and local camera cues. A conditional denoiser trained with a diffusion loss performs only three reverse steps at inference, balancing speed and accuracy while enforcing physically plausible confidence estimates.
Temporal consistency is encouraged via Temporal Self‑Distillation (TSD). The Mamba state predicts the next‑frame BEV embedding; an L1 loss between this prediction and the stop‑gradient of the actual next embedding forces smooth evolution without explicit motion labels.
All components are jointly optimized with a composite loss comprising classification, regression, IoU, uncertainty regularization, geometric constraint, diffusion, and temporal consistency terms.
Extensive experiments on the nuScenes benchmark demonstrate state‑of‑the‑art mAP and NDS, with notable robustness to simulated calibration errors (±2°) and severe LiDAR sparsity (up to 50 % point removal). The hybrid SSM‑window design yields linear computational complexity, enabling real‑time inference (>15 FPS) on a single GPU.
In summary, MambaFusion advances multimodal 3D perception by (i) providing an efficient linear‑time global context mechanism, (ii) dynamically re‑weighting sensor contributions based on learned spatial reliability and uncertainty, (iii) enforcing physical plausibility through graph reasoning and diffusion‑based confidence refinement, and (iv) stabilizing predictions across time via self‑distillation. The combination of these innovations results in a robust, interpretable, and deployment‑ready detection pipeline for autonomous driving.