LangGS-SLAM: Real-Time Language-Feature Gaussian Splatting SLAM

In this paper, we propose a RGB-D SLAM system that reconstructs a language-aligned dense feature field while sustaining low-latency tracking and mapping. First, we introduce a Top-K Rendering pipeline, a high-throughput and semantic-distortion-free method for efficiently rendering high-dimensional feature maps. To address the resulting semantic-geometric discrepancy and mitigate the memory consumption, we further design a multi-criteria map management strategy that prunes redundant or inconsistent Gaussians while preserving scene integrity. Finally, a hybrid field optimization framework jointly refines the geometric and semantic fields under real-time constraints by decoupling their optimization frequencies according to field characteristics. The proposed system achieves superior geometric fidelity compared to geometric-only baselines and comparable semantic fidelity to offline approaches while operating at 15 FPS. Our results demonstrate that online SLAM with dense, uncompressed language-aligned feature fields is both feasible and effective, bridging the gap between 3D perception and language-based reasoning.

💡 Research Summary

LangGS‑SLAM presents a novel real‑time SLAM framework that builds a dense, language‑aligned feature field directly from RGB‑D streams and pre‑computed vision‑language model (VLM) embeddings such as CLIP or LSeg. The system tackles three core challenges that have prevented online semantic SLAM from scaling to high‑dimensional, open‑vocabulary features: (1) the computational burden of rendering millions of Gaussians with 512‑dimensional vectors, (2) semantic distortion caused by conventional alpha‑blending, and (3) the memory overhead of storing a feature vector per Gaussian.

To address (1) and (2), the authors introduce a Top‑K rendering pipeline for the semantic field. During the standard alpha‑blending pass for geometry (color and depth), the contribution weight of each Gaussian is recorded. For each pixel, only the K Gaussians with the highest weights are selected; their weights are renormalized and used to linearly combine the associated VLM feature vectors. This reduces the rendering complexity from O(N·D) to O(K·D) (where N is the number of Gaussians and D the feature dimension) and prevents mixing of unrelated surface semantics, which would otherwise produce ambiguous feature vectors. A custom CUDA kernel implements both the geometric pass and the Top‑K semantic pass, re‑using the index and weight information to achieve deterministic, high‑throughput execution.

Memory and redundancy are handled by a two‑stage multi‑criteria map management strategy. First, a semantic‑geometric consistency pruning step identifies Gaussians that are rarely selected by Top‑K (low semantic contribution) and evaluates their maximum geometric contribution across all keyframes. A survival probability proportional to this geometric score is computed, and a weighted sampling retains a predefined fraction of these candidates, ensuring that geometrically important Gaussians are not discarded even if their semantic influence is weak. Second, during map updates, a redundancy‑aware insertion mechanism re‑uses the nearest‑neighbor distances computed by the G‑ICP tracker: if a newly proposed Gaussian lies within a threshold of an existing one, insertion is suppressed, preventing unnecessary growth of the map without extra computation.

Optimization is performed in a hybrid fashion. All Gaussian parameters—geometric (position, covariance, opacity, color) and semantic (VLM feature)—are optimized by minimizing a weighted sum of a geometric loss (L1 color and depth consistency) and a semantic loss (L1 distance between rendered and ground‑truth VLM feature maps). Because the semantic field is smoother and depends on a stable geometry, the authors update geometry at every iteration while updating semantics at a lower frequency (e.g., every 5–10 frames). This decoupling reduces redundant computation and accelerates convergence of both fields.

The backbone of the system is GS‑ICP SLAM, which provides fast pose estimation and an efficient Gaussian initialization scheme. When a frame’s overlap with the current map falls below a threshold, it becomes a keyframe; new Gaussians are seeded from depth data and VLM feature maps. The entire pipeline runs at approximately 15 FPS on a high‑end GPU (RTX 4090), achieving geometric reconstruction errors lower than geometric‑only baselines and semantic quality (mIoU) comparable to offline methods such as LERF or Feature‑3DGS. Memory usage is reduced by about 30 % thanks to the pruning strategy, while still storing full 512‑dim vectors per Gaussian (≈2 KB per Gaussian).

Qualitative experiments demonstrate open‑vocabulary 3D queries (“find the red chair”, “list objects on the desk”) directly on the reconstructed map, showcasing the potential for language‑driven perception in robotics, AR/VR, and embodied AI. Limitations include sensitivity to the chosen K value (smaller K may lose fine details) and the lack of feature compression, which could become a bottleneck for very large scenes. Future work is suggested to explore learned compression decoders, adaptive K selection, and tighter integration with LLMs for online semantic refinement.

In summary, LangGS‑SLAM delivers a practical solution for real‑time dense SLAM with uncompressed, language‑aligned feature fields, bridging the gap between 3D geometric reconstruction and high‑level language reasoning while maintaining real‑time performance.