DM3D: Deformable Mamba via Offset-Guided Gaussian Sequencing for Point Cloud Understanding

State Space Models (SSMs) demonstrate significant potential for long-sequence modeling, but their reliance on input order conflicts with the irregular nature of point clouds. Existing approaches often rely on predefined serialization strategies, which cannot adjust based on diverse geometric structures. To overcome this limitation, we propose \textbf{DM3D}, a deformable Mamba architecture for point cloud understanding. Specifically, DM3D introduces an offset-guided Gaussian sequencing mechanism that unifies local resampling and global reordering within a deformable scan. The Gaussian-based KNN Resampling (GKR) enhances structural awareness by adaptively reorganizing neighboring points, while the Gaussian-based Differentiable Reordering (GDR) enables end-to-end optimization of serialization order. Furthermore, a Tri-Path Frequency Fusion module enhances feature complementarity and reduces aliasing. Together, these components enable structure-adaptive serialization of point clouds. Extensive experiments on benchmark datasets show that DM3D achieves state-of-the-art performance in classification, few-shot learning, and part segmentation, demonstrating that adaptive serialization effectively unlocks the potential of SSMs for point cloud understanding. The code will be released at https://github.com/L1277471578/DM3D.

💡 Research Summary

The paper introduces DM3D, a novel architecture that integrates a deformable scanning mechanism into the Mamba family of State Space Models (SSMs) for 3‑D point cloud understanding. Traditional SSMs excel at long‑sequence modeling but require a fixed input order, which conflicts with the unordered nature of point clouds. Existing serialization methods (e.g., space‑filling curves, MLP projections, voxel or graph unfoldings) are static and cannot adapt to the diverse geometric structures present in point clouds, leading to loss of local continuity and sub‑optimal feature learning.

DM3D addresses this gap by proposing an offset‑guided Gaussian sequencing mechanism that jointly performs local resampling and global reordering in a deformable scan. The system first extracts local groups via farthest point sampling (FPS) and K‑nearest neighbor (KNN) grouping, then encodes each group with a lightweight PointNet. A classification token is prepended and the initial token sequence is generated using a Hilbert curve.

The core Deformable Mamba Block (DMB) contains three parallel SSM branches: a standard forward SSM (F‑SSM), a channel‑flip backward SSM (C‑SSM) for stability, and a new Deformable SSM (D‑SSM). D‑SSM receives spatial and sequential offsets predicted by an OffsetNet. The offsets are bounded by tanh and split into Δp (3‑D coordinate offsets) and Δt (scalar sequence offsets).

For spatial adaptation, Gaussian‑based KNN Resampling (GKR) uses the offset points p′ = p + Δp to query Kᵣ nearest neighbors in the original point set. Features are interpolated with Gaussian weights W(d;σₛ)=exp(−d²/2σₛ²), ensuring that nearby points contribute more strongly while distant points fade smoothly. No bias terms are added, as point clouds lack a regular grid.

For sequence adaptation, Gaussian‑based Differentiable Reordering (GDR) treats sorting as a continuous probabilistic mapping. The base index vector I (derived from the Hilbert ordering) is shifted by Δt to obtain a continuous position s. A pairwise distance matrix (s − J) (where J =