Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems

Bongard Problems (BPs) provide a challenging testbed for abstract visual reasoning (AVR), requiring models to identify visual concepts fromjust a few examples and describe them in natural language. Early BP benchmarks featured synthetic black-and-white drawings, which might not fully capture the complexity of real-world scenes. Subsequent BP datasets employed real-world images, albeit the represented concepts are identifiable from high-level image features, reducing the task complexity. Differently, the recently released Bongard-RWR dataset aimed at representing abstract concepts formulated in the original BPs using fine-grained real-world images. Its manual construction, however, limited the dataset size to just $60$ instances, constraining evaluation robustness. In this work, we introduce Bongard-RWR+, a BP dataset composed of $5,400$ instances that represent original BP abstract concepts using real-world-like images generated via a vision language model (VLM) pipeline. Building on Bongard-RWR, we employ Pixtral-12B to describe manually curated images and generate new descriptions aligned with the underlying concepts, use Flux.1-dev to synthesize images from these descriptions, and manually verify that the generated images faithfully reflect the intended concepts. We evaluate state-of-the-art VLMs across diverse BP formulations, including binary and multiclass classification, as well as textual answer generation. Our findings reveal that while VLMs can recognize coarse-grained visual concepts, they consistently struggle with discerning fine-grained concepts, highlighting limitations in their reasoning capabilities.

💡 Research Summary

Bongard problems (BPs) are a classic few‑shot visual reasoning benchmark: each problem consists of two panels, each containing six images that share a hidden abstract rule, and a solver must infer the rule and often verbalize it. Early BP datasets used synthetic black‑and‑white drawings, which are far from the complexity of real‑world scenes. Later datasets (Bongard‑HOI, Bongard‑OpenWorld) introduced natural images, but the concepts they encode are high‑level and can be identified from coarse visual cues, making the task considerably easier for modern vision‑language models (VLMs). The recently released Bongard‑RWR attempted to bridge this gap by representing the original abstract concepts with real‑world photographs, yet its manual construction limited the corpus to only 60 instances, hampering robust evaluation.

This paper presents Bongard‑RWR+, a dramatically scaled dataset containing 5 400 BP instances that preserve the original abstract concepts while using realistic‑looking images generated automatically. The authors design a semi‑automated pipeline that leverages state‑of‑the‑art VLM components: (1) an image‑to‑text model (Pixtral‑12B) produces a pair of captions for each source image – a positive caption that faithfully describes the image and a negative caption that steers the next stage away from the opposite concept; (2) a text‑to‑text model expands each positive caption into 15 diverse paraphrases, preserving the underlying rule; (3) a text‑to‑image diffusion model (Flux.1‑dev) renders candidate images from each (positive, negative) caption pair; (4) human annotators verify that the generated images indeed embody the intended side‑concept without leaking elements of the opposite side.

From the verified pool, the pipeline selects subsets that maximize intra‑side visual diversity (by minimizing cosine similarity of ViT‑L/14 embeddings) and constructs 10 left‑side and 10 right‑side panels per original problem. Pairing each left with each right yields 100 new BPs per source, resulting in 5 400 matrices covering 49 distinct abstract concepts (derived from 54 original matrices; six were discarded due to generation difficulties). In addition to the main dataset, the authors release variants: a grayscale version (Bongard‑RWR+/GS) to isolate the effect of color, and versions with varying numbers of demonstration images per side (P = 2…6) to study few‑shot scaling.

The benchmark supports six task formulations: (a) Image‑to‑Side (I1S) – assign a single test image to the correct side; (b) Images‑to‑Sides (I2S) – assign a pair of test images; (c) Description‑to‑Side (D1S) and (d) Descriptions‑to‑Sides (D2S) – use generated captions as inputs; (e) Concept Selection (CS) – choose the correct natural‑language rule from a candidate list; (f) Concept Generation (CG) – produce a free‑form textual description of the rule.

The authors evaluate a suite of contemporary VLMs (including CLIP‑ViT, BLIP‑2, LLaVA, and several multimodal LLM hybrids) across all tasks. Results reveal a consistent pattern: models reliably identify coarse‑grained concepts such as “vertical”, “circular”, or “red objects”, achieving 70‑80 % accuracy on binary classification. However, when the rule requires fine‑grained relational reasoning—e.g., “all arrows point in the same direction” versus “arrows point in different directions”—performance collapses to below 35 % accuracy, and text‑generation models often produce irrelevant attributes (color, background) instead of the intended rule. The grayscale variant shows negligible performance change, confirming that color is not a decisive cue for these abstract tasks. Increasing the number of demonstration images per side yields modest gains, but the improvement saturates quickly, indicating that the bottleneck lies in relational reasoning rather than data quantity.

The analysis highlights two central insights. First, despite massive pre‑training on image‑text pairs, current VLMs excel at recognizing individual visual attributes but lack mechanisms to integrate multiple images and infer relational constraints. Second, the semi‑automated generation pipeline, while dramatically more scalable than manual curation, still depends on human verification to guarantee concept fidelity, and the generated images inherit biases from the underlying diffusion model (e.g., over‑representation of certain object categories).

The paper concludes by outlining future directions: (1) developing automatic consistency checks (e.g., cycle‑consistency between generated images and captions) to reduce human labor; (2) designing architectures that explicitly model inter‑image relations, such as graph‑based transformers or relational attention modules; (3) expanding the dataset with adversarially hard examples that target known VLM weaknesses; and (4) investigating curriculum‑style training that gradually introduces finer relational concepts. Bongard‑RWR+ thus provides a challenging, scalable benchmark for evaluating and advancing the abstract visual reasoning capabilities of next‑generation multimodal AI systems.