Geospatial-Reasoning-Driven Vocabulary-Agnostic Remote Sensing Semantic Segmentation

Open-vocabulary semantic segmentation has emerged as a promising research direction in remote sensing, enabling the recognition of diverse land-cover types beyond pre-defined category sets. However, existing methods predominantly rely on the passive mapping of visual features and textual embeddings. This ``appearance-based" paradigm lacks geospatial contextual awareness, leading to severe semantic ambiguity and misclassification when encountering land-cover classes with similar spectral features but distinct semantic attributes. To address this, we propose a Geospatial Reasoning Chain-of-Thought (GR-CoT) framework designed to enhance the scene understanding capabilities of Multimodal Large Language Models (MLLMs), thereby guiding open-vocabulary segmentation models toward precise mapping. The framework comprises two collaborative components: an offline knowledge distillation stream and an online instance reasoning stream. The offline stream establishes fine-grained category interpretation standards to resolve semantic conflicts between similar land-cover types. During online inference, the framework executes a sequential reasoning process involving macro-scenario anchoring, visual feature decoupling, and knowledge-driven decision synthesis. This process generates an image-adaptive vocabulary that guides downstream models to achieve pixel-level alignment with correct geographical semantics. Extensive experiments on the LoveDA and GID5 benchmarks demonstrate the superiority of our approach.

💡 Research Summary

The paper tackles the persistent problem of semantic ambiguity in open‑vocabulary remote sensing semantic segmentation, where spectrally similar land‑cover classes (e.g., plastic greenhouses vs. industrial buildings) are frequently confused by existing “appearance‑based” methods. To overcome this limitation, the authors propose a Geospatial Reasoning Chain‑of‑Thought (GR‑CoT) framework that leverages the logical reasoning capabilities of Multimodal Large Language Models (MLLMs) to generate image‑adaptive vocabularies that guide downstream segmentation.

GR‑CoT consists of two collaborative streams. The offline knowledge‑distillation stream first prompts an MLLM with each class from a global category pool C to produce detailed, multi‑dimensional descriptions covering morphology, spectral‑spatial attributes, and spatial exclusivity. For pairs of classes that are visually alike but semantically distinct (e.g., agricultural greenhouses vs. industrial structures, barren land vs. cultivated fields), a fine‑grained discrimination step defines rigorous inter‑class relationships. The result is a set of Category Interpretation Standards S = {(c_i, D_i)} that encode expert‑level geographic knowledge.

The online instance‑reasoning stream operates at inference time. Given an input image I, it first performs macro‑scenario anchoring f_anchor(I) to infer a global context G (urban, rural, industrial, etc.), which serves as a geographical prior. Next, visual feature decoupling f_decouple(I, G) extracts a collection of discrete visual attributes A = {a_j}, each describing texture, reflectance, shape, or fine‑grained sub‑category cues. Finally, a knowledge‑driven decision synthesis step verifies each candidate class c_i against G, A, and the standards S via a verification function. Only classes that satisfy all constraints are retained, forming an image‑adaptive vocabulary V_adaptive.

During segmentation, V_adaptive replaces the full textual candidate set in the pixel‑to‑text alignment equation M(x, y) = argmax_{c_j ∈ V_adaptive} ⟨F_v(x, y), E_t(c_j)⟩, where F_v denotes the visual feature map and E_t the text encoder embedding. By narrowing the search space to context‑consistent categories, the framework dramatically reduces cross‑category misclassifications.

Experimental evaluation on two challenging benchmarks—LoveDA and GID‑5—demonstrates the effectiveness of GR‑CoT. On LoveDA, the method achieves a mean Intersection‑over‑Union (mIoU) of 41.39 % and overall accuracy (OA) of 59.93 %, outperforming the strongest baselines (CA‑T‑Seg and RSKT‑Seg) by +7.2 %p and +8.2 %p respectively. Notably, the agricultural class IoU rises from ~46 % (baseline) to 61.19 %, and background IoU improves from near zero to 10.57 %, illustrating the framework’s ability to prune false positives in fragmented regions. On GID‑5, GR‑CoT reaches mIoU 45.34 % and OA 63.34 %, again surpassing baselines, with especially large gains in farmland (65.32 % IoU) and meadow (26.60 % IoU) where spectral confusion is common.

Ablation studies confirm the contribution of each component. Using only the offline knowledge (no online reasoning) raises category accuracy from 11.19 % to 45.12 %, while adding macro‑scenario anchoring and visual feature decoupling further lifts accuracy to 64.85 % and mIoU to 45.34 %. These results validate that both the expert knowledge base and the dynamic, context‑aware reasoning chain are essential.

The authors acknowledge limitations: the reliance on large MLLMs incurs substantial computational cost for knowledge distillation and online inference; prompt engineering for knowledge extraction remains manual; and the current system assumes a predefined category pool, limiting true “open‑ended” vocabulary expansion. Future work may explore lightweight LLMs, automated prompt generation, and continual learning to broaden applicability.

In summary, GR‑CoT redefines open‑vocabulary remote sensing segmentation by shifting from passive visual‑semantic matching to active geospatial reasoning. By integrating offline expert knowledge with online context‑driven inference, it produces image‑specific vocabularies that align pixel‑level predictions with high‑level geographic semantics, achieving state‑of‑the‑art performance on benchmark datasets. This work highlights the critical role of geospatial logic in advancing robust, scalable scene understanding for remote sensing applications.