Imandra CodeLogician: Neuro-Symbolic Reasoning for Precise Analysis of Software Logic

Large Language Models (LLMs) have shown strong performance on code understanding tasks, yet they fundamentally lack the ability to perform precise, exhaustive mathematical reasoning about program behavior. Existing benchmarks either focus on mathematical proof automation, largely disconnected from real-world software, or on engineering tasks that do not require semantic rigor. We present CodeLogician, a neurosymbolic agent for precise analysis of software logic, integrated with ImandraX, an industrial automated reasoning engine deployed in financial markets and safety-critical systems. Unlike prior approaches that use formal methods primarily to validate LLM outputs, CodeLogician uses LLMs to construct explicit formal models of software systems, enabling automated reasoning to answer rich semantic questions beyond binary verification outcomes. To rigorously evaluate mathematical reasoning about software logic, we introduce code-logic-bench, a benchmark targeting the middle ground between theorem proving and software engineering benchmarks. It measures reasoning correctness about program state spaces, control flow, coverage constraints, and edge cases, with ground truth defined via formal modeling and region decomposition. Comparing LLM-only reasoning against LLMs augmented with CodeLogician, formal augmentation yields substantial improvements, closing a 41-47 percentage point gap in reasoning accuracy. These results demonstrate that neurosymbolic integration is essential for scaling program analysis toward rigorous, autonomous software understanding.

💡 Research Summary

The paper introduces CodeLogician, a neuro‑symbolic framework that tightly couples large language models (LLMs) with the industrial automated reasoning engine ImandraX to enable precise, exhaustive analysis of software logic. Rather than using formal methods merely as a post‑hoc filter for LLM‑generated code or proofs, CodeLogician employs LLMs as translators that automatically convert source code into formal models expressed in the Imandra Modeling Language (IML), a pure functional language enriched with specification and verification constructs. These IML models are then handed to ImandraX, which performs theorem proving, region decomposition, and systematic test generation. Region decomposition partitions the program’s state space into mathematically defined regions, allowing ImandraX to reason about coverage, decision boundaries, and edge cases with provable guarantees.

The framework is deliberately reasoner‑agnostic: while ImandraX is the primary backend, the architecture abstracts the reasoning layer so that other provers (Coq, Isabelle, Z3, etc.) can be swapped in without redesign. CodeLogician offers multiple entry points—a command‑line interface, a text‑based UI, a VS Code extension, and a Python RemoteGraph API—making formal analysis accessible throughout typical development pipelines. The LLM‑driven auto‑formalization agent also annotates uncertain constructs as opaque functions, axioms, or approximations, thereby preserving the limits of the formal model.

To evaluate the system, the authors create a new benchmark, code‑logic‑bench, which fills the gap between pure mathematical proof benchmarks and engineering‑focused code tasks. The benchmark consists of real‑world financial and safety‑critical code snippets, each accompanied by ground‑truth specifications derived from formal modeling and automated region decomposition. Metrics include state‑space estimation accuracy, outcome precision, direction accuracy, coverage completeness, control‑flow understanding, edge‑case detection, and decision‑boundary clarity.

Experiments compare several state‑of‑the‑art LLMs (GPT‑4, Claude‑2, Gemini‑1.5) used in isolation versus the same models augmented with CodeLogician. LLM‑only approaches achieve roughly 38 % average correctness, struggling especially with exhaustive state‑space reasoning and boundary identification. When coupled with CodeLogician, accuracy jumps to over 85 % across all metrics, closing a 41–47 percentage‑point gap. Detailed case studies—LSE GTT order expiry, London Stock Exchange fee schedule, and a multilateral netting engine—demonstrate how the system automatically formalizes code, decomposes state spaces, generates high‑coverage test suites, and verifies subtle floating‑point precision issues that would be missed by heuristic methods.

The key insight is that LLMs excel at translating ambiguous, high‑level code into a formal representation, while automated reasoning engines provide the exhaustive logical guarantees that LLMs lack. By separating translation from verification, CodeLogician delivers a scalable, principled approach to software analysis that can be extended to other reasoning backends and programming languages. The authors conclude that neuro‑symbolic integration is essential for advancing autonomous, rigorous software understanding and outline future work on broader language support, richer human‑AI collaboration interfaces, and deeper integration with industrial CI/CD pipelines.