On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation

The emergence of large-scale pre-trained point cloud models has significantly advanced 3D scene understanding, but adapting these models to specific downstream tasks typically demands full fine-tuning, incurring high computational and storage costs. Parameter-efficient fine-tuning (PEFT) techniques, successful in natural language processing and 2D vision tasks, would underperform when naively applied to 3D point cloud models due to significant geometric and spatial distribution shifts. Existing PEFT methods commonly treat points as orderless tokens, neglecting important local spatial structures and global geometric contexts in 3D modeling. To bridge this gap, we introduce the Geometric Encoding Mixer (GEM), a novel geometry-aware PEFT module specifically designed for 3D point cloud transformers. GEM explicitly integrates fine-grained local positional encodings with a lightweight latent attention mechanism to capture comprehensive global context, thereby effectively addressing the spatial and geometric distribution mismatch. Extensive experiments demonstrate that GEM achieves performance comparable to or sometimes even exceeding full fine-tuning, while only updating 1.6% of the model’s parameters, fewer than other PEFT methods. With significantly reduced training time and memory requirements, our approach thus sets a new benchmark for efficient, scalable, and geometry-aware fine-tuning of large-scale 3D point cloud models. Code is available at https://github.com/LiyaoTang/GEM.

💡 Research Summary

The paper tackles the costly adaptation of large pre‑trained point‑cloud transformers for 3D scene segmentation by introducing a geometry‑aware parameter‑efficient fine‑tuning (PEFT) module called the Geometry Encoding Mixer (GEM). Existing PEFT techniques such as adapters, LoRA, and prompt tuning have proven effective in natural language processing and 2D vision, but they underperform on point‑cloud data because they treat points as unordered tokens and ignore the strong spatial and geometric cues inherent in 3D scenes. Moreover, modern 3D transformers rely on local attention to keep computational complexity manageable, which further limits the ability of conventional PEFT methods to capture global scene context.

GEM addresses these challenges with two complementary adapters. The Spatial Adapter refines the pre‑trained positional encoding by applying a lightweight 3‑D convolutional bottleneck over each point’s local neighborhood. This module aggregates nearby points within a voxel grid, projects the aggregated features to a low‑dimensional space, applies a non‑linear activation, and projects back, thereby enriching each point’s representation with fine‑grained geometric detail while adding only a tiny fraction of parameters. The Context Adapter introduces a small set of learned latent tokens that act as global context vectors. These tokens interact with the full point set through an efficient cross‑attention mechanism, effectively bypassing the locality constraint of the backbone’s attention layers and providing a compact summary of scene‑wide geometry and semantics.

Both adapters are inserted as residual branches, so the original backbone weights remain frozen. The total number of trainable parameters is roughly 1.6 % of the full model, yet GEM consistently matches or exceeds the performance of full fine‑tuning across several large‑scale indoor (S3DIS, ScanNet) and outdoor (SemanticKITTI) benchmarks. Quantitatively, GEM improves mean Intersection‑over‑Union (mIoU) by 4–6 % over prior PEFT baselines and reduces training time by more than 30 % while cutting GPU memory consumption by around 40 %. The method is compatible with various point‑cloud transformer architectures (e.g., PT, PTv3, PointTransformer), demonstrating its generality.

The authors also provide a thorough analysis of why standard PEFT methods fail on 3D data: they either adapt at the per‑point level without considering spatial structure, or they inject a few static global tokens that cannot capture the diverse scene contexts present in point clouds. By explicitly modeling both local geometry (through the Spatial Adapter) and global context (through the Context Adapter), GEM establishes a new paradigm for geometry‑aware PEFT in 3D vision. The paper releases code and pretrained models, facilitating reproducibility and practical deployment.