LaVPR: Benchmarking Language and Vision for Place Recognition

Visual Place Recognition (VPR) often fails under extreme environmental changes and perceptual aliasing. Furthermore, standard systems cannot perform “blind” localization from verbal descriptions alone, a capability needed for applications such as emergency response. To address these challenges, we introduce LaVPR, a large-scale benchmark that extends existing VPR datasets with over 650,000 rich natural-language descriptions. Using LaVPR, we investigate two paradigms: Multi-Modal Fusion for enhanced robustness and Cross-Modal Retrieval for language-based localization. Our results show that language descriptions yield consistent gains in visually degraded conditions, with the most significant impact on smaller backbones. Notably, adding language allows compact models to rival the performance of much larger vision-only architectures. For cross-modal retrieval, we establish a baseline using Low-Rank Adaptation (LoRA) and Multi-Similarity loss, which substantially outperforms standard contrastive methods across vision-language models. Ultimately, LaVPR enables a new class of localization systems that are both resilient to real-world stochasticity and practical for resource-constrained deployment. Our dataset and code are available at https://github.com/oferidan1/LaVPR.

💡 Research Summary

Visual Place Recognition (VPR) suffers from severe performance degradation under extreme environmental changes and perceptual aliasing. Moreover, conventional VPR systems cannot perform “blind” localization based solely on verbal descriptions—a capability required for emergency response, human‑robot interaction, and forensic geolocation. To address these gaps, the authors introduce LaVPR, a large‑scale benchmark that augments several widely used VPR datasets (GSV‑Cities, Pitts30K, AmsterTime, MSLS) with a total of 651,865 high‑quality natural‑language captions. Captions are generated with Gemini 2.5 Flash, emphasizing permanent architectural elements and spatial relationships while minimizing transient details. A three‑stage validation pipeline—automatic entity extraction (Phi‑3.5‑mini), segmentation grounding (SAM‑3), and visual‑language verification (Qwen2‑VL)—achieves 91 % recall and 22 % precision, after which a human‑in‑the‑loop review removes the remaining ~1 % hallucinations. The final dataset exhibits an average of 52.7 words per caption and 23.3 nouns, with roughly 29 % of samples containing prominent signage or text.

The benchmark enables systematic study of two paradigms: (1) Multi‑Modal Fusion (Vision + Language) and (2) Cross‑Modal Retrieval (Language → Vision). For fusion, the authors keep the visual encoder (E_v) and textual encoder (E_t) frozen and explore four late‑fusion strategies: simple concatenation, projection‑addition, a shallow MLP, and Adaptive Score Fusion (ADS). ADS learns modality‑specific similarity weights via a small MLP, allowing the system to emphasize the more reliable modality per query‑reference pair. Additionally, a Learned Language Pooling (LLP) module applies self‑attention over the textual hidden states to produce a context‑aware representation. Experiments across multiple backbones (NetVLAD, CosPlace, EigenPlaces, MixVPR, SALAD, CricaVPR) and datasets show that ADS and simple concatenation consistently improve Recall@1, with gains ranging from +0.3 % to +0.5 % on average. Notably, compact backbones such as ViT‑S, when combined with language, achieve performance comparable to much larger models, highlighting language as a universal regularizer that mitigates visual degradation. Specialized subsets targeting heavy blur or adverse weather confirm that linguistic cues provide the most benefit when visual signals are severely corrupted.

For cross‑modal retrieval, the authors demonstrate that zero‑shot CLIP and standard contrastive fine‑tuning fail to deliver precise place identification. They propose a parameter‑efficient alignment method that couples Low‑Rank Adaptation (LoRA) with the Multi‑Similarity (MS) loss. LoRA updates only a small set of low‑rank matrices in the vision‑language model, preserving the bulk of pretrained knowledge while adapting to the place‑recognition domain. The MS loss simultaneously pushes positive pairs together and pushes negatives apart with adaptive margins, leading to a more discriminative shared latent space. This combination yields Recall@1 improvements of 6 %–12 % over baseline contrastive methods across all test splits, with the most pronounced gains on the MSLS‑Blur and MSLS‑Weather subsets where visual cues are unreliable.

Overall, LaVPR establishes that natural language can serve as a robust, high‑level semantic anchor for VPR. Language not only boosts performance under challenging visual conditions but also enables “blind” localization, opening avenues for resource‑constrained deployment (e.g., edge devices, drones) where large visual backbones are impractical. The benchmark’s open‑source release (code and data) provides a standardized platform for future research on multimodal place recognition, cross‑modal retrieval, and language‑driven robotic navigation. Limitations include residual hallucination rates in the caption generation pipeline and the current focus on English descriptions; future work should aim at improving automated validation, extending to multilingual captions, and integrating additional modalities such as depth or LiDAR for a truly sensor‑agnostic localization system.