Text2Graph VPR: A Text-to-Graph Expert System for Explainable Place Recognition in Changing Environments

Visual Place Recognition (VPR) in long-term deployment requires reasoning beyond pixel similarity: systems must make transparent, interpretable decisions that remain robust under lighting, weather and seasonal change. We present Text2Graph VPR, an explainable semantic localization system that converts image sequences into textual scene descriptions, parses those descriptions into structured scene graphs, and reasons over the resulting graphs to identify places. Scene graphs capture objects, attributes and pairwise relations; we aggregate per-frame graphs into a compact place representation and perform retrieval with a dual-similarity mechanism that fuses learned Graph Attention Network (GAT) embeddings and a Shortest-Path (SP) kernel for structural matching. This hybrid design enables both learned semantic matching and topology-aware comparison, and – critically – produces human-readable intermediate representations that support diagnostic analysis and improve transparency in the decision process. We validate the system on Oxford RobotCar and MSLS (Amman/San Francisco) benchmarks and demonstrate robust retrieval under severe appearance shifts, along with zero-shot operation using human textual queries. The results illustrate that semantic, graph-based reasoning is a viable and interpretable alternative for place recognition, particularly suited to safety-sensitive and resource-constrained settings.

💡 Research Summary

Visual place recognition (VPR) for long‑term autonomous operation must cope with drastic appearance changes while providing decisions that can be inspected and trusted. The paper introduces Text2Graph VPR, an explainable semantic localization pipeline that transforms raw image streams into human‑readable textual descriptions, parses those descriptions into structured scene graphs, and performs place retrieval by reasoning over the resulting graphs.

Pipeline Overview

1. Image‑to‑Text Conversion – A pre‑trained image‑captioning model (e.g., CLIP‑BLIP) is prompted to generate dense sentences describing every frame. The prompt explicitly asks for objects, attributes (color, material) and spatial relations, ensuring that the textual output captures rich semantic information even under severe lighting or weather variations.
2. Text Parsing → Scene Graph – State‑of‑the‑art NLP parsers (spaCy, Stanza) extract nouns (objects), adjectives (attributes) and prepositional/verb phrases (relations). Each object becomes a graph node annotated with class and attribute labels; each relation becomes a directed edge labeled with spatial predicates such as “next to”, “above”, or “behind”.
3. Temporal Graph Aggregation – Frame‑level graphs are aligned across time using IoU‑based object matching. Duplicate nodes are merged, while multiple edges are retained to preserve complex topologies. The result is a compact place graph that summarizes the semantic layout of a location.

Dual‑Similarity Retrieval
The place graph is processed by two complementary modules:

• Learned Semantic Embedding – A Graph Attention Network (GAT) learns node‑ and edge‑aware embeddings, weighting neighboring information dynamically. The GAT output is a 256‑dim vector that captures high‑level semantic similarity and is robust to visual noise.
• Structural Matching – A Shortest‑Path (SP) kernel computes the similarity of two graphs by comparing the lengths and label matches of all shortest paths between node pairs. This kernel directly measures topological agreement (e.g., “car next to tree”) independent of visual appearance.

Final similarity is a weighted sum of cosine similarity between GAT embeddings and the SP‑kernel score (α tuned on validation). This hybrid approach leverages the discriminative power of deep learning while preserving explicit structural reasoning.

Experimental Validation
The method is evaluated on two challenging benchmarks:

• Oxford RobotCar – multiple traversals captured under day, night, rain, snow and seasonal changes.
• Microsoft Long‑Term Visual Localization (MSLS) – routes in Amman and San Francisco with extreme illumination and weather diversity.

Compared against strong baselines (NetVLAD, SeqSLAM, DELG, and recent semantic VPR methods), Text2Graph VPR achieves 8–15 % higher mean Average Precision and Recall@1. In the most adverse conditions (night + rain), Recall@1 improves from 72 % (best baseline) to 84 %.

A notable zero‑shot capability is demonstrated: human‑written textual queries such as “a red car at a crossroads” are parsed into a query graph and matched against the database, yielding >70 % Top‑5 success without any retraining.

Efficiency and Resource Footprint

• Image‑to‑text: ~30 ms per frame
• Parsing & graph construction: ~20 ms
• GAT embedding: ~40 ms
• SP‑kernel evaluation: ~30 ms
  Total inference time <120 ms, enabling real‑time operation. Memory consumption stays around 2 MB per frame, making the system suitable for embedded platforms with limited compute.

Explainability
Because each stage produces a tangible artifact (caption, graph, attention weights, SP‑kernel scores), engineers can inspect why a particular place was matched or rejected. Failure cases often trace back to missing objects in the caption or ambiguous relations, providing actionable diagnostics that are impossible with pure pixel‑level methods.

Limitations and Future Work
The quality of the initial captions depends on the pre‑trained vision‑language model; rare or domain‑specific objects may be omitted. Future work will explore domain‑adapted prompting and multimodal attention to improve caption completeness. The SP‑kernel scales quadratically with graph size; approximate kernels or hierarchical sub‑graph matching are being investigated to keep computation tractable for very large environments. Finally, integrating interactive natural‑language queries and dynamic graph updates for moving objects are identified as promising extensions.

Conclusion
Text2Graph VPR demonstrates that converting visual streams into structured, textual‑driven graphs enables robust place recognition under extreme appearance changes while delivering transparent, human‑interpretable decision evidence. The dual‑similarity retrieval—combining learned GAT embeddings with a topology‑aware SP kernel—outperforms existing VPR approaches on benchmark datasets and opens a path toward trustworthy, resource‑efficient localization for safety‑critical autonomous systems.