Geo3DVQA: Evaluating Vision-Language Models for 3D Geospatial Reasoning from Aerial Imagery

Three-dimensional geospatial analysis is critical for applications in urban planning, climate adaptation, and environmental assessment. However, current methodologies depend on costly, specialized sensors, such as LiDAR and multispectral sensors, which restrict global accessibility. Additionally, existing sensor-based and rule-driven methods struggle with tasks requiring the integration of multiple 3D cues, handling diverse queries, and providing interpretable reasoning. We present Geo3DVQA, a comprehensive benchmark that evaluates vision-language models (VLMs) in height-aware 3D geospatial reasoning from RGB imagery alone. Unlike conventional sensor-based frameworks, Geo3DVQA emphasizes realistic scenarios integrating elevation, sky view factors, and land cover patterns. The benchmark comprises 110k curated question-answer pairs across 16 task categories, including single-feature inference, multi-feature reasoning, and application-level analysis. Through a systematic evaluation of ten state-of-the-art VLMs, we reveal fundamental limitations in RGB-to-3D spatial reasoning. Our results further show that domain-specific instruction tuning consistently enhances model performance across all task categories, including height-aware and open-ended, application-oriented reasoning. Geo3DVQA provides a unified, interpretable framework for evaluating RGB-based 3D geospatial reasoning and identifies key challenges and opportunities for scalable 3D spatial analysis. The code and data are available at https://github.com/mm1129/Geo3DVQA.

💡 Research Summary

Geo3DVQA introduces a novel benchmark designed to evaluate the capability of vision‑language models (VLMs) to perform three‑dimensional (3D) geospatial reasoning using only aerial RGB imagery. The authors argue that while 3D geospatial analysis is essential for urban planning, climate adaptation, and environmental assessment, existing approaches rely heavily on expensive sensors such as LiDAR and multispectral cameras, limiting accessibility worldwide. To address this gap, Geo3DVQA compiles 110 000 question‑answer (QA) pairs across 16 task categories, organized into three tiers of increasing complexity: (1) single‑feature inference (e.g., estimating Sky View Factor, land‑cover type, or average building height), (2) multi‑feature reasoning (integrating SVF, building density, terrain flatness, etc., into composite metrics like sky‑visibility score or spatial openness), and (3) application‑level free‑form analysis (e.g., renewable‑energy potential, water‑accumulation risk, urban development recommendations).

The dataset is built on the GeoNRW collection, which provides high‑resolution RGB orthophotos, LiDAR‑derived digital surface models (DSM), and semantic land‑cover maps for North Rhine‑Westphalia, Germany. Ground‑truth values for SVF, elevation, and land‑cover are computed automatically from these multimodal references using established tools (e.g., the UMEP toolkit for SVF via ray‑casting). Question templates are generated programmatically, and distractor answers are sampled to ensure balanced label distributions. Human annotators verify a subset to guarantee scientific validity. Strict geographic splits separate training and test regions, preventing spatial leakage and enabling assessment of generalization to unseen areas.

For evaluation, ten state‑of‑the‑art VLMs are benchmarked, including open‑source models (LLaVA‑OneVision, InternVL‑3‑8B, Qwen2.5‑VL‑7B/13B) and commercial APIs (GPT‑4o, Gemini‑2.5‑Flash, GPT‑4‑mini). Two fine‑tuning regimes are explored: a modest 10 K short‑answer QA set and a larger 100 K set, each mixed with 1 K free‑form QA examples. Models receive only the RGB image as input; all other modalities are hidden during inference. Evaluation metrics comprise per‑category accuracy for short answers, Jaccard similarity for multi‑label tasks (threshold 0.8), tolerance‑based scoring for numeric height/SVF estimates, and a 1‑5 rubric for free‑form responses that assesses observation quality, conclusion relevance, and application suitability.

Results reveal that, out‑of‑the‑box, VLMs achieve modest performance (≈30 % overall short‑answer accuracy). GPT‑4o and Gemini‑2.5‑Flash are the strongest among the zero‑shot models, yet they still struggle with continuous‑value estimation and multi‑feature integration. Domain‑specific instruction tuning dramatically improves outcomes: a Qwen2.5‑VL‑7B model fine‑tuned on the Geo3DVQA data reaches nearly 50 % overall accuracy, a gain of over 20 percentage points, with the most pronounced improvements in Tier 2 (multi‑feature) and Tier 3 (application) tasks. This demonstrates that VLMs can learn height‑aware reasoning when provided with targeted supervision, but the underlying limitation remains that RGB‑only inputs are intrinsically ill‑posed for precise 3D reconstruction. The models rely on indirect cues such as shadows, occlusions, and texture gradients to make coarse height distinctions, which suffices for bin‑level or relative comparisons but not for fine‑grained absolute measurements.

The authors highlight several key insights: (1) current VLMs lack robust 3D spatial reasoning despite impressive 2D performance; (2) multi‑modal ground truth is essential for generating reliable QA pairs, but the benchmark deliberately hides this information during inference to test true RGB‑based reasoning; (3) instruction tuning with domain‑specific prompts and data is a practical pathway to bridge the performance gap; (4) the three‑tier taxonomy provides a structured way to diagnose where models fail—whether at basic feature extraction, feature integration, or higher‑level application reasoning.

Geo3DVQA’s contributions are threefold: (i) it delivers the first large‑scale, height‑aware VQA benchmark for remote sensing that operates solely on RGB imagery; (ii) it offers a comprehensive evaluation protocol covering single‑feature, multi‑feature, and real‑world application queries; (iii) it demonstrates that domain‑specific fine‑tuning can substantially improve performance, suggesting a roadmap for future research. The authors propose extending the benchmark with additional modalities (e.g., multispectral or LiDAR) to study multimodal fusion, exploring more sophisticated prompt engineering for continuous value estimation, and applying the benchmark in operational settings such as urban planning dashboards or disaster‑response tools.

In summary, Geo3DVQA provides a rigorous, reproducible platform to assess and advance the ability of vision‑language models to infer three‑dimensional geospatial information from ubiquitous RGB aerial imagery, revealing both current limitations and promising avenues for improvement.