Lang3D-XL: Language Embedded 3D Gaussians for Large-scale Scenes

Fig. 1. Language Embedded 3D Gaussians for Large-scale Scenes. Given Internet collections of images depicting large-scale scenes (left), we augment a 3D Gaussian representation with a learnable semantic bottleneck (center). Our approach enables interactive text-based virtual exploration (right), enabling users to zoom-in and view semantic regions of interests, such as the spires and lintels * in the Milano Catedral depicted above. * A spire refers to a tall, slender, pointed structure on top of a roof of a building or tower, and a lintel is a horizontal structural element that spans openings such as doors and windows.

Embedding a language field in a 3D representation enables richer semantic understanding of spatial environments by linking geometry with descriptive meaning. This allows for a more intuitive human-computer interaction, enabling querying or editing scenes using natural language, and could potentially improve tasks like scene retrieval, navigation, and multimodal reasoning. While such capabilities could be transformative, in particular for large-scale scenes, we find that recent feature distillation approaches cannot effectively learn over massive Internet data due to challenges in semantic feature misalignment and inefficiency in memory and runtime. To this end, we propose a novel approach to address these challenges. First, we intr

…(Content truncated for length.)

📄 Read Full PDF on ArXiv

This content is AI-processed based on open access ArXiv data.