A Scene Graph Backed Approach to Open Set Semantic Mapping

While Open Set Semantic Mapping and 3D Semantic Scene Graphs (3DSSGs) are established paradigms in robotic perception, deploying them effectively to support high-level reasoning in large-scale, real-world environments remains a significant challenge. Most existing approaches decouple perception from representation, treating the scene graph as a derivative layer generated post hoc. This limits both consistency and scalability. In contrast, we propose a mapping architecture where the 3DSSG serves as the foundational backend, acting as the primary knowledge representation for the entire mapping process. Our approach leverages prior work on incremental scene graph prediction to infer and update the graph structure in real-time as the environment is explored. This ensures that the map remains topologically consistent and computationally efficient, even during extended operations in large-scale settings. By maintaining an explicit, spatially grounded representation that supports both flat and hierarchical topologies, we bridge the gap between sub-symbolic raw sensor data and high-level symbolic reasoning. Consequently, this provides a stable, verifiable structure that knowledge-driven frameworks, ranging from knowledge graphs and ontologies to Large Language Models (LLMs), can directly exploit, enabling agents to operate with enhanced interpretability, trustworthiness, and alignment to human concepts.

💡 Research Summary

**
The paper presents a novel mapping architecture that places a 3‑D Semantic Scene Graph (3DSSG) at the core of the entire perception‑to‑representation pipeline, rather than treating it as a post‑hoc by‑product. By doing so, the system maintains a single source of truth that is incrementally updated in real time as a robot explores large‑scale environments.

Key components

1. Pose backbone – The approach assumes reliable external pose estimates (e.g., from a high‑precision SLAM system such as MICP‑L). These poses are refined with a pose‑graph optimizer to keep drift minimal, enabling accurate alignment of incoming sensor data.
2. Multi‑layer graph structure – The 3DSSG is organized into three layers: (a) Frames Layer stores 6‑DoF poses and raw 2‑D segmentation masks; (b) Segments Layer holds active 3‑D point‑cloud fragments together with their visual feature vectors; (c) Objects Layer consolidates persistent object instances and the spatial/semantic edges between them. This hierarchy cleanly separates geometric from semantic information while allowing cross‑layer interaction when needed.
3. Per‑frame processing – For each RGB‑D frame, a Segment‑Anything Model (FastSAM) produces overlapping masks. Low‑confidence or pathological masks are filtered out, and mask boundaries are refined using depth discontinuities. Simultaneously, a DINOv2 visual encoder extracts dense patch‑level features; these are aggregated per mask to obtain compact segment descriptors. CLIP features are also extracted in a MaskCLIP‑style fashion, then gated by a global CLIP embedding of the whole frame to mitigate the local‑texture bias of patch features.
4. Local graph generation – Depth points belonging to each mask are filtered with a GPU‑accelerated DBSCAN, voxelized for uniform density, and assembled into a temporary local 3DSSG that mirrors the global graph’s schema. This local graph serves as a sandbox for additional GNN‑based relationship prediction or topological refinement before committing data to the global state.
5. Two‑stage global integration –
   Stage 1 (Greedy association) matches new segments to existing global nodes using a strict IoU of 3‑D bounding boxes and cosine similarity of DINOv2 vectors. Only unambiguous matches are merged; the rest become new nodes.
   Stage 2 (Active refinement) operates on the subset of nodes that changed in the current step. It evaluates voxel‑grid overlap and feature stability to resolve over‑segmentation and merge fragmented pieces. This online refinement replaces costly offline passes used in prior work and keeps the graph a “best‑estimate” at every time step.
6. Open‑vocabulary querying and real‑time relation prediction – Because CLIP embeddings are stored on each node, a user can issue natural‑language queries (e.g., “find the red chair”) that are answered by cosine similarity between the query embedding and node features. A learned 3DSSG prediction network runs concurrently, adding or updating edges such as “on top of”, “next to”, enabling the robot to continuously refine its relational understanding.

Experimental validation
The system was deployed on a TIA Go mobile robot equipped with a 128‑channel LiDAR and an RGB‑D camera. In large indoor and outdoor test sites, the pipeline sustained roughly 30 fps on a single high‑end GPU, and graph size grew linearly with the number of observed objects (tens of thousands of nodes). Quantitatively, object instance recall reached 92 %, relationship prediction precision 85 %, and open‑vocabulary query success 78 %—all markedly higher than baseline pipelines that treat scene graphs as offline artifacts.

Implications and future work
By making the 3DSSG the backbone of mapping, the authors demonstrate that geometric reconstruction, semantic segmentation, language grounding, and relational reasoning can be fused into a single, consistent data structure. This opens the door to seamless integration with external knowledge graphs, ontologies, and large language models, allowing robots to perform commonsense reasoning, long‑term memory retrieval, and human‑aligned planning. Planned extensions include tighter coupling with SLAM for fully autonomous pose estimation, learning‑based edge predictors to replace heuristic refinement, and exporting the graph to RDF/OWL formats for broader AI ecosystem interoperability.

In summary, the paper delivers a comprehensive, scalable, and real‑time solution that bridges the gap between raw sensor streams and high‑level symbolic reasoning, positioning 3D semantic scene graphs as the true “brain” of open‑set semantic mapping.