OBSR: Open Benchmark for Spatial Representations

GeoAI is evolving rapidly, fueled by diverse geospatial datasets like traffic patterns, environmental data, and crowdsourced OpenStreetMap (OSM) information. While sophisticated AI models are being developed, existing benchmarks are often concentrated on single tasks and restricted to a single modality. As such, progress in GeoAI is limited by the lack of a standardized, multi-task, modality-agnostic benchmark for their systematic evaluation. This paper introduces a novel benchmark designed to assess the performance, accuracy, and efficiency of geospatial embedders. Our benchmark is modality-agnostic and comprises 7 distinct datasets from diverse cities across three continents, ensuring generalizability and mitigating demographic biases. It allows for the evaluation of GeoAI embedders on various phenomena that exhibit underlying geographic processes. Furthermore, we establish a simple and intuitive task-oriented model baselines, providing a crucial reference point for comparing more complex solutions.

💡 Research Summary

The paper introduces OBSR (Open Benchmark for Spatial Representations), a comprehensive, modality‑agnostic benchmark designed to evaluate geospatial embedding models across multiple tasks and data types. Recognizing that existing geospatial benchmarks are typically single‑task and single‑modality, the authors assemble seven publicly available datasets that span diverse domains—housing (Airbnb listings, King County house sales), public safety (Chicago, Philadelphia, and San Francisco crime reports), and mobility (Porto taxi GPS traces, Beijing Geolife trajectories). These datasets cover six cities on three continents, thereby reducing geographic and socioeconomic bias.

OBSR’s design follows four core principles: (1) breadth of datasets and tasks, (2) reproducibility through clearly defined train/test splits, (3) granularity via multiple H3 hexagonal resolutions to capture spatial dependencies at different scales, and (4) accessibility through a HuggingFace repository and integration with the SRAI library, which standardizes loading, preprocessing, and evaluation.

The benchmark defines two major categories of downstream tasks: region‑based prediction (e.g., estimating average house price, crime density per hexagon) and trajectory‑based prediction (e.g., next‑location inference, mobility pattern classification). For each task, standard metrics such as RMSE, F1‑score, inference latency, and memory consumption are reported, allowing simultaneous assessment of accuracy and efficiency.

To provide a reference point, the authors implement a simple baseline that relies only on basic map‑derived statistics (e.g., mean values per H3 cell). This baseline is intentionally lightweight, serving as a yardstick against which more sophisticated Geospatial Foundation Models (GeoFMs) can be compared. Experimental results show that while the baseline performs reasonably on single‑modality, single‑task settings, it lags substantially behind advanced models in the multi‑task, multi‑modality scenario, highlighting OBSR’s ability to expose gaps in generalization and transfer learning.

All data, code, and evaluation scripts are released under permissive licenses, encouraging community contributions and future extensions (new datasets, tasks, or modalities). By offering a unified, reproducible, and extensible platform, OBSR aims to become the de‑facto standard for systematic GeoAI model assessment, fostering more transparent and comparable research progress across the field.