LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understanding

Applying Gaussian Splatting to perception tasks for 3D scene understanding is becoming increasingly popular. Most existing works primarily focus on rendering 2D feature maps from novel viewpoints, which leads to an imprecise 3D language field with outlier languages, ultimately failing to align objects in 3D space. By utilizing masked images for feature extraction, these approaches also lack essential contextual information, leading to inaccurate feature representation. To this end, we propose a Language-Embedded Surface Field (LangSurf), which accurately aligns the 3D language fields with the surface of objects, facilitating precise 2D and 3D segmentation with text query, widely expanding the downstream tasks such as removal and editing. The core of LangSurf is a joint training strategy that flattens the language Gaussian on the object surfaces using geometry supervision and contrastive losses to assign accurate language features to the Gaussians of objects. In addition, we also introduce the Hierarchical-Context Awareness Module to extract features at the image level for contextual information then perform hierarchical mask pooling using masks segmented by SAM to obtain fine-grained language features in different hierarchies. Extensive experiments on open-vocabulary 2D and 3D semantic segmentation demonstrate that LangSurf outperforms the previous state-of-the-art method LangSplat by a large margin. As shown in Fig. 1, our method is capable of segmenting objects in 3D space, thus boosting the effectiveness of our approach in instance recognition, removal, and editing, which is also supported by comprehensive experiments. https://langsurf.github.io.

💡 Research Summary

LangSurf introduces a novel framework that tightly integrates natural‑language semantics with 3D Gaussian Splatting (3DGS) by embedding language features directly onto the surfaces of objects. Existing open‑vocabulary 3D scene understanding methods such as LangSplat and LERF primarily render 2‑D feature maps from novel viewpoints and then use CLIP‑SAM pipelines to obtain a “3‑D language field”. This pipeline suffers from two major drawbacks: (1) the use of local masks or sliding windows discards global contextual cues, leading to poor semantic representations for low‑texture regions (walls, floors) or complex objects that are split into many parts; and (2) the language features are not explicitly constrained to lie on the true geometric surfaces, causing spatial inconsistency and outlier Gaussians that degrade downstream tasks such as 3‑D querying, segmentation, and editing.

To overcome these limitations, LangSurf proposes two key innovations. First, the Hierarchical‑Context Awareness Module extracts pixel‑level visual‑language features for the whole image using a pretrained image encoder (e.g., a Vision Transformer). It then leverages the Segment‑Anything Model (SAM) to obtain multi‑scale masks (small, medium, large) and performs hierarchical mask‑average pooling. This yields context‑aware features for each mask that combine global scene information with local object details, greatly improving semantic consistency for texture‑poor or structurally complex regions. The pooled features are compressed through an auto‑encoder into a low‑dimensional 3‑channel latent map, reducing memory usage while preserving discriminative power.

Second, LangSurf adopts a three‑stage joint training strategy to construct a language‑embedded surface field:

1. RGB‑Only Stage – Standard 3DGS training with RGB reconstruction loss (L_rgb) and a Gaussian flattening loss (L_flat) that compresses Gaussians onto object planes.

2. Geometry + Semantic Joint Stage – Geometry regularization (L_geo) enforces multi‑view normal consistency, aligning Gaussians with true surfaces. Semantic alignment loss (L_sem) minimizes L2 distance between rendered language features (F_lang) and the latent context‑aware features (L′_lang). A Semantic Grouping loss (L_sg) encourages features inside the same SAM mask to be close, preserving intra‑object consistency while sharpening inter‑object boundaries. Finally, a Spatial‑Aware Semantic Supervision (L_s3d) based on KL‑divergence penalizes outlier language Gaussians that drift away from surfaces, further tightening spatial alignment.

3. Instance‑Aware Stage – After the language field is well‑trained, each Gaussian receives an instance embedding (f_ins). A contrastive loss maximizes distances between different object instances while keeping intra‑instance features compact, enabling precise instance‑level text queries, removal, and editing.

The overall loss is a weighted sum of the above components, and training proceeds sequentially from coarse geometry to fine semantic and instance refinement.

Experimental Evaluation
LangSurf is evaluated on the LERF and ScanNet datasets for open‑vocabulary 2‑D and 3‑D semantic segmentation. Metrics include mean accuracy (mAcc) and mean Intersection‑over‑Union (mIoU) for both 2‑D and 3‑D tasks. On LERF, LangSurf achieves mAcc 84.57 % and mIoU 60.02 %, surpassing LangSplat’s 73.57 % / 51.90 % by a large margin. On ScanNet, average 3‑D IoU improves from 36.52 % (LangSplat) to 47.91 % (LangSurf). Qualitative results (Fig. 1, 3) demonstrate that the language field aligns tightly with object surfaces, leading to cleaner object removal and editing with minimal artifacts.

Technical Significance

1. Surface‑Aligned Language Embedding – By flattening language Gaussians onto geometry, LangSurf eliminates the spatial gap that plagued prior methods, achieving consistent semantics across viewpoints.
2. Hierarchical Context Integration – The mask‑pooling strategy simultaneously captures global scene context and fine‑grained object cues, addressing the long‑standing issue of low‑texture region ambiguity.
3. Progressive Multi‑Stage Training – Starting from RGB reconstruction, then adding geometry and semantic constraints, and finally instance discrimination, the method incrementally refines both shape and meaning, preserving the real‑time rendering speed of 3DGS.

Limitations and Future Work
LangSurf relies heavily on SAM’s mask quality; erroneous masks can propagate to the language field. Memory consumption, while mitigated by the auto‑encoder, still grows with the number of Gaussians for large scenes. Future research may explore mask‑free context extraction, adaptive Gaussian pruning, or integration with diffusion‑based 3‑D generation pipelines to further improve scalability and robustness.

Conclusion
LangSurf presents a compelling solution for open‑vocabulary 3‑D scene understanding by embedding language directly onto object surfaces and enriching those embeddings with hierarchical contextual information. The method delivers substantial gains in both quantitative benchmarks and practical downstream applications such as text‑driven 3‑D segmentation, object removal, and editing, positioning it as a strong baseline for future multimodal 3‑D perception systems.