Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation

Multimodal large language models (MLLMs) have achieved strong performance on perception-oriented tasks, yet their ability to perform mathematical spatial reasoning, defined as the capacity to parse and manipulate two- and three-dimensional relations, remains unclear. Humans easily solve textbook-style spatial reasoning problems with over 95% accuracy, but we find that most leading MLLMs fail to reach even 60% on the same tasks. This striking gap highlights spatial reasoning as a fundamental weakness of current models. To investigate this gap, we present MathSpatial, a unified framework for evaluating and improving spatial reasoning in MLLMs. MathSpatial includes three complementary components: (i) MathSpatial-Bench, a benchmark of 2K problems across three categories and eleven subtypes, designed to isolate reasoning difficulty from perceptual noise; (ii) MathSpatial-Corpus, a training dataset of 8K additional problems with verified solutions; and (iii) MathSpatial-SRT, which models reasoning as structured traces composed of three atomic operations–Correlate, Constrain, and Infer. Experiments show that fine-tuning Qwen2.5-VL-7B on MathSpatial achieves competitive accuracy while reducing tokens by 25%. MathSpatial provides the first large-scale resource that disentangles perception from reasoning, enabling precise measurement and comprehensive understanding of mathematical spatial reasoning in MLLMs.

💡 Research Summary

This paper investigates whether multimodal large language models (MLLMs) truly possess the ability to perform mathematical spatial reasoning – the capacity to parse, relate, and manipulate two‑ and three‑dimensional geometric relations. While recent MLLMs have achieved impressive results on perception‑heavy tasks such as image captioning, visual question answering, and action recognition, their competence on pure spatial reasoning remains uncertain. Human participants solve textbook‑style spatial problems with over 95 % accuracy, yet the authors find that leading MLLMs (including GPT‑4V, LLaVA‑1.5, and other state‑of‑the‑art systems) consistently fall below 60 % on the same set, exposing a substantial capability gap.

To systematically study this gap, the authors introduce MathSpatial, a unified framework comprising three complementary components:

1. MathSpatial‑Bench – a benchmark of 2,000 carefully curated geometry problems. Each problem is rendered with minimal background and texture to eliminate perceptual confounds, and is categorized into three high‑level groups (Holistic Recognition, Generative Inference, Abstract Deduction) and eleven fine‑grained sub‑types (e.g., multi‑view correspondence, missing‑view completion, geometric property calculation). Human accuracy on this benchmark exceeds 95 %, whereas all evaluated MLLMs stay under 60 %.

2. MathSpatial‑Corpus – a training corpus of 8,000 additional educational geometry problems sourced from public textbooks and exam banks spanning primary to high‑school levels. The pipeline for corpus creation involves (a) large‑scale collection, (b) standardization into a unified schema (image, question, choices, answer, solution), (c) rigorous de‑duplication using MD5 hashes, GPT‑4 vision similarity, and semantic filtering, (d) geometric consistency checks (length‑width‑height correspondence, orthographic projection rules), and (e) solution verification by graduate‑level annotators with dual‑review. The final dataset is bilingual (Chinese and English) and includes human‑verified solutions.

3. MathSpatial‑SRT (Structured Reasoning Traces) – a novel supervision paradigm that decomposes spatial problem solving into three atomic operations:

   • Correlate – establishing correspondences across multiple views or between diagram elements,
   • Constrain – applying geometric and projection constraints to enforce consistency,
   • Infer – deriving final attributes or answers from the constrained representation.

   Using GPT‑4o, the authors automatically generate structured reasoning traces for every problem, then employ human cross‑validation to reduce trace errors to under 10 %. These traces serve as intermediate supervision, making the model’s reasoning process transparent and diagnosable, unlike conventional end‑to‑end chain‑of‑thought (CoT) approaches.

Experimental Findings
The authors fine‑tune the open‑source multimodal model Qwen2.5‑VL‑7B on MathSpatial‑Corpus with the SRT supervision. Compared to several closed‑source baselines, the fine‑tuned model achieves competitive accuracy (approximately 3–5 % higher) while reducing the average number of generated tokens by about 25 %. Error analysis reveals that most failures occur during the Correlate stage (incorrect view matching) and the Constrain stage (misapplication of geometric rules). The structured traces enable precise pinpointing of these failure modes, suggesting that future work should focus on stronger multi‑view alignment modules and more explicit encoding of geometric constraints.

Contributions

• Introduces the first large‑scale, perception‑free benchmark for mathematical spatial reasoning, allowing clean separation of visual perception and logical reasoning.
• Provides a substantial, high‑quality training corpus with bilingual support and verified solutions, filling a critical data scarcity gap.
• Proposes an interpretable reasoning framework (SRT) that supplies intermediate supervision, facilitating both model performance gains and transparent diagnostics.

Implications and Future Directions
MathSpatial establishes a solid foundation for evaluating and improving spatial cognition in multimodal models. The authors envision extending the SRT paradigm to other domains such as physics simulation, robotic manipulation, and embodied AI, where spatial reasoning is essential. Moreover, integrating human‑in‑the‑loop feedback with SRT could enable real‑time error correction and continual learning. By disentangling perception from reasoning and supplying structured supervision, MathSpatial paves the way toward MLLMs that approach human‑level spatial understanding.