Thinking in Structures: Evaluating Spatial Intelligence through Reasoning on Constrained Manifolds

Spatial intelligence is crucial for vision–language models (VLMs) in the physical world, yet many benchmarks evaluate largely unconstrained scenes where models can exploit 2D shortcuts. We introduce SSI-Bench, a VQA benchmark for spatial reasoning on constrained manifolds, built from complex real-world 3D structures whose feasible configurations are tightly governed by geometric, topological, and physical constraints. SSI-Bench contains 1,000 ranking questions spanning geometric and topological reasoning and requiring a diverse repertoire of compositional spatial operations, such as mental rotation, cross-sectional inference, occlusion reasoning, and force-path reasoning. It is created via a fully human-centered pipeline: ten researchers spent over 400 hours curating images, annotating structural components, and designing questions to minimize pixel-level cues. Evaluating 31 widely used VLMs reveals a large gap to humans: the best open-source model achieves 22.2% accuracy and the strongest closed-source model reaches 33.6%, while humans score 91.6%. Encouraging models to think yields only marginal gains, and error analysis points to failures in structural grounding and constraint-consistent 3D reasoning. Project page: https://ssi-bench.github.io.

💡 Research Summary

The paper introduces SSI‑Bench, a novel visual‑question‑answering benchmark specifically designed to evaluate the spatial intelligence of vision‑language models (VLMs) under the regime of Constrained‑Manifold Spatial Reasoning (CMSR). Existing spatial benchmarks largely focus on unconstrained scenes where models can exploit 2‑D correlations, appearance priors, or dataset‑specific regularities without constructing a physically consistent 3‑D representation. In contrast, SSI‑Bench targets real‑world engineering structures—such as space frames, steel towers, cable‑stayed bridges, timber trusses, and pipeline systems—whose feasible configurations are tightly governed by geometric, topological, and physics‑based constraints.

The authors formalize CMSR by representing a structural scene as a tuple (V, E, G, A) where V and E are nodes and members, G encodes continuous geometric degrees of freedom, and A captures discrete attributes. Feasibility constraints are expressed as equality (c) and inequality (h) functions that define a constrained manifold M. Given one or more images x, a model must infer a latent 3‑D state s that lies on M and then evaluate a task‑specific criterion function fτ(s, c) for each candidate c. The benchmark frames each query as a ranking problem: a set of 3 or 4 candidates is presented, and the model must output the permutation that orders them according to fτ. This ranking formulation yields a quantifiable, comparable metric because the manifold constraints narrow the space of plausible 3‑D states, stabilizing the target relations across plausible interpretations.

Construction of SSI‑Bench follows a fully human‑centered pipeline. Ten researchers with interdisciplinary expertise spent over 400 hours reviewing roughly 20 000 structure‑related images from royalty‑free sources (Unsplash, Pexels, Pixabay) and their own photography, ultimately curating more than 2 000 candidates that cover a broad spectrum of engineering forms. For each image, annotators manually labeled nodes, members, connectivity graphs, and geometric attributes using Label Studio. They also recorded the correct ascending order for each task and provided precise localization polygons for the referenced components. To prevent shortcuts, each candidate is visualized in a separate image with distinct highlight colors, and ties are resolved by a deterministic rule (smaller index first).

SSI‑Bench comprises 1 000 multiple‑choice ranking questions divided into two families: Geometric (Ground Height, Ground Angle, Dimension, Relative Distance, Area, Volume) and Topological (Hop Distance, Cycle Length, etc.). Geometric tasks are mostly at the member level with four candidates; topological tasks operate at the group level with three candidates. A Multi‑View subset supplies two viewpoints per question, requiring cross‑view correspondence to a reference member, thereby testing the model’s ability to integrate information across views and handle occlusions. Difficulty labels are assigned to every question based on human adjudication.

The benchmark is evaluated on 31 VLMs, including 10 proprietary models from four families and 21 open‑source models (e.g., LLaVA‑1.5‑13B, MiniGPT‑4, InstructBLIP). Models receive the same prompts and images; a “think step‑by‑step” variant is also tested. The best open‑source model attains 22.2 % accuracy, while the strongest closed‑source model reaches 33.6 %, compared with a human average of 91.6 %. Encouraging models to “think” yields only marginal gains (≈2–3 % absolute).

Error analysis reveals two dominant failure modes. First, structural grounding is weak: models often misidentify or completely miss the target members, especially under heavy occlusion or complex intersecting geometry, preventing the construction of an accurate connectivity graph. Second, constraint‑consistent 3‑D reasoning is lacking; inferred configurations frequently violate equality constraints (e.g., mismatched member lengths) or inequality constraints (e.g., impossible support conditions), leading to incorrect rankings. Tasks that require compositional operations—mental rotation of complex shapes, cross‑section inference, occlusion reasoning, and force‑path reasoning—show the steepest performance drops.

The authors argue that SSI‑Bench fills a critical gap by providing a rigorous, constraint‑aware evaluation of spatial intelligence, moving beyond superficial 2‑D cues toward genuine 3‑D reasoning. They suggest future directions: (1) integrating multi‑modal pre‑training that includes point clouds or CAD data to improve structural grounding; (2) embedding constraint‑solving modules within VLMs to enforce feasibility during inference; (3) expanding the benchmark with richer physical constraints (material properties, dynamic loads); and (4) exploring interactive “think‑aloud” prompting strategies to better elicit stepwise reasoning. Overall, SSI‑Bench establishes a challenging testbed that can drive the next generation of VLMs toward true spatial cognition in the physical world.