Visual Model Checking: Graph-Based Inference of Visual Routines for Image Retrieval

Information retrieval lies at the foundation of the modern digital industry. While natural language search has seen dramatic progress in recent years largely driven by embedding-based models and large-scale pretraining, the field still faces significant challenges. Specifically, queries that involve complex relationships, object compositions, or precise constraints such as identities, counts and proportions often remain unresolved or unreliable within current frameworks. In this paper, we propose a novel framework that integrates formal verification into deep learning-based image retrieval through a synergistic combination of graph-based verification methods and neural code generation. Our approach aims to support open-vocabulary natural language queries while producing results that are both trustworthy and verifiable. By grounding retrieval results in a system of formal reasoning, we move beyond the ambiguity and approximation that often characterize vector representations. Instead of accepting uncertainty as a given, our framework explicitly verifies each atomic truth in the user query against the retrieved content. This allows us to not only return matching results, but also to identify and mark which specific constraints are satisfied and which remain unmet, thereby offering a more transparent and accountable retrieval process while boosting the results of the most popular embedding-based approaches.

💡 Research Summary

The paper introduces a novel framework that brings formal verification techniques into the domain of image retrieval, addressing the persistent shortcomings of purely embedding‑based methods when faced with complex natural‑language queries. The authors propose a “visual model checking” pipeline that converts a free‑form textual query into a structured logical specification, synthesizes executable visual routines for each atomic predicate, and then verifies candidate images against these routines.

Pipeline Overview

1. Query Parsing – A system parser (P) transforms the user query (q) into a graph (\phi) consisting of subject‑predicate‑object triples. This graph serves as a formal specification of the desired visual scene.
2. Routine Synthesis – For each triple (\phi_i), a synthesizer model (M) (implemented with a large language model such as Microsoft Phi‑4) generates a Python routine (\pi_i). The routine encodes calls to pre‑defined visual APIs (object detectors, relationship classifiers, OCR, etc.) that can test the truth of the triple on an image.
3. Verification – Each routine (\pi_i) is executed on a candidate image (v), returning a Boolean value. An image fully satisfies the query only if all (\pi_i(v)=\text{True}). Otherwise, the proportion of satisfied triples yields a “truth score”: (\text{Score}(v)=\frac{#\text{VerifiedTriplets}}{#\text{TotalTriplets}}).
4. Re‑ranking – The truth score is combined with a baseline embedding‑based ranking (e.g., CLIP) to produce a hybrid score: (\text{ReRankScore}_i = (K-i) \times \text{Score}(v_i)), where (K) is the number of top candidates and (i) is the position in the original ranking. This re‑ranking boosts images that satisfy more constraints, even if their raw embedding similarity is lower.

Theoretical Considerations
The authors acknowledge the classic state‑explosion problem of model checking. By treating each triple independently, they avoid constructing a full world model; instead, global truth is approximated as the conjunction of local Boolean results. This mirrors the “global truth = conjunction of local truths” principle in traditional verification, allowing scalable verification across large image collections.

Experimental Setup
Evaluation uses the MS‑COCO Captions validation split. To isolate difficult queries, the set is split into COCO‑Easy (top 25 % of CLIP retrieval performance) and COCO‑Hard (bottom 25 %). Recall@1, @5, and @10 are reported for several baselines (CLIP ViT‑B/H, SigLIP, ALIGN, BEIT‑3) and for the proposed method alone and in combination with those baselines. The system runs on a heterogeneous GPU cluster, with the pipeline distributed across parsing, routine synthesis, and visual inference stages.

Results

• On the full COCO set, the proposed method alone achieves recall comparable to state‑of‑the‑art zero‑shot models (e.g., Recall@1 ≈ 0.43 vs. CLIP 0.49).
• On the Hard split, the hybrid approaches (e.g., “ours + ALIGN”) markedly outperform pure embeddings, especially for Recall@1 (0.261 vs. 0.152 for CLIP).
• The improvement is most pronounced for queries requiring counting, text recognition, or multi‑object relationships, confirming that the visual routines can capture constraints that embeddings ignore.
• Qualitative examples illustrate successful detection of “person riding a horse” or “two giraffes touching while a third eats from a tree,” while failure cases often stem from API limitations or erroneous code generated by the language model.

Limitations

1. Scalability – The number of generated routines grows linearly with the number of triples; for queries with many constraints, computational cost can become prohibitive.
2. API Dependence – The approach relies on a fixed set of visual modules; extending to novel concepts (e.g., fine‑grained actions, emotions) requires adding new APIs and retraining the synthesizer.
3. LLM‑Generated Code Reliability – Language‑model synthesis can produce syntactically invalid or semantically incorrect code, necessitating additional validation or fallback mechanisms.

Conclusions and Future Work
The paper demonstrates that integrating formal verification into image retrieval yields a system that is both more trustworthy and more capable of handling complex, compositional queries. By providing explicit per‑constraint satisfaction signals, the framework offers transparency that pure embedding methods lack. Future directions include developing more efficient routine management (e.g., caching, hierarchical verification), expanding the library of visual APIs, and tighter integration with large multimodal models to reduce reliance on handcrafted modules. The work opens a promising avenue toward retrieval systems that can not only retrieve but also prove that retrieved images meet user‑specified criteria.