Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

Extracting structured procedural knowledge from unstructured business documents is a critical yet unresolved bottleneck in process automation. While prior work has focused on extracting linear action flows from instructional texts, such as recipes, it has insufficiently addressed the complex logical structures, including conditional branching and parallel execution, that are pervasive in real-world regulatory and administrative documents. Furthermore, existing benchmarks are limited by simplistic schemas and shallow logical dependencies, restricting progress toward logic-aware large language models.To bridge this Logic Gap, we introduce BREX, a carefully curated benchmark comprising 409 real-world business documents and 2,855 expert-annotated rules. Unlike prior datasets centered on narrow service scenarios, BREX spans over 30 vertical domains, covering scientific, industrial, administrative, and financial regulations. We further propose ExIde, a structure-aware reasoning framework that investigates five distinct prompting strategies, ranging from implicit semantic alignment to executable grounding via pseudo-code generation. This enables explicit modeling of rule dependencies and provides an out-of-the-box framework for different business customers without finetuning their own large language models. We benchmark ExIde using 13 state-of-the-art large language models. Our extensive evaluation reveals that executable grounding serves as a superior inductive bias, significantly outperforming standard prompts in rule extraction. In addition, reasoning-optimized models demonstrate a distinct advantage in tracing long-range and non-linear rule dependencies compared to standard instruction-tuned models.

💡 Research Summary

The paper tackles the “Logic Gap” – the disconnect between free‑form natural‑language regulations and the executable, condition‑dependent control flows required by automated business systems. To bridge this gap, the authors introduce two major contributions: the BREX benchmark and the ExIde framework.

BREX (Business Rule Extraction Benchmark)
BREX is a cross‑domain dataset specifically designed for business rule flow modeling. It comprises 409 real‑world business documents drawn from over 30 verticals (scientific research, industrial manufacturing, administrative approvals, financial compliance, etc.) and 2,855 expert‑annotated atomic rules. Each rule is formalized as a condition‑action pair, where the condition is a structured triple ⟨Slot Type, Logical Operator, Reference Value⟩ and the action is a free‑text operation. Rules are linked via explicit dependency relations of three types: Sequential (must execute before another), Conditional (branching based on outcomes), and Parallel (must be executed concurrently). The dataset construction involved (1) collecting authentic documents, (2) augmenting under‑represented logical patterns with carefully filtered synthetic texts generated by Gemini 2.5 Pro, (3) expert annotation by three domain specialists, and (4) a multi‑stage verification process involving a separate trio of experts. Quality assessments report high readability, accuracy, and clarity (average ICC = 0.892) and near‑perfect inter‑annotator agreement on the projected NER representation (Fleiss’ κ = 0.901). Over 30 % of the rules participate in conditional or parallel dependencies, confirming the prevalence of non‑linear logic in real regulations.

ExIde (Executable Ideation) Framework
ExIde adopts a decompose‑and‑reason strategy that separates rule extraction from global dependency reasoning. In Stage I, five distinct prompting strategies (P1–P5) are evaluated, all sharing the same output schema but differing in the inductive bias they provide:

• P1 – Implicit Semantic Alignment: The model first generates a natural‑language explanation of the business logic, then maps it to the structured output without explicit alignment.
• P2 – Explicit Traceability: Each explanatory sentence is explicitly linked to a rule (“This sentence corresponds to the business rule: …”), reducing hallucinations.
• P3 – Clarified Contextual Input: The input field is explicitly labeled as “Business Text Input” to prime domain awareness.
• P4 – Logic‑Aware Definition Injection: Detailed constraints for logical operators (e.g., set inclusion vs. equality) are injected via a “Note” section, guiding the model on multi‑value slots.
• P5 – Executable Grounding (Pseudo‑Code): The most novel approach; the model first translates the document into a simple pseudo‑code language (e.g., if condition: execute_action()) using a small set of primitives, then extracts the final condition‑action rules from this code. This intermediate representation forces early resolution of nested conditions and control flow, acting as a strong inductive bias for logic‑intensive extraction.

All prompts employ chain‑of‑thought reasoning, but only the final structured outputs are evaluated.

In Stage II, ExIde reconstructs a global dependency graph G = (V, E) where vertices are the extracted rules and edges encode Sequential, Conditional, and Parallel relations. The graph is validated against the gold‑standard annotations.

Experimental Evaluation
Thirteen state‑of‑the‑art LLMs (including GPT‑4, Claude‑2, Llama‑2‑70B, Gemini‑1.5, etc.) were tested across the five prompting strategies. Metrics comprised rule extraction F1, dependency‑graph reconstruction accuracy, and a specialized long‑range dependency tracing score. Key findings:

1. Executable Grounding Wins: Prompt 5 consistently outperformed the others, achieving an average F1 improvement of 7–9 percentage points. The advantage was most pronounced on documents with deep nesting and parallel constraints, confirming that pseudo‑code grounding supplies a powerful inductive bias.
2. Reasoning‑Optimized Models Excel: Models explicitly trained for logical reasoning (e.g., Claude‑2‑Sonnet, GPT‑4‑Turbo) showed a 12 %+ boost in long‑range dependency tracing compared to standard instruction‑tuned models, highlighting the importance of inherent reasoning capabilities.
3. Ablation Insights: Adding explicit traceability (P2) modestly reduced hallucinations but did not match the gains from executable grounding. Context clarification (P3) and definition injection (P4) yielded small but consistent improvements, especially for multi‑value slot handling.

Implications and Future Work
The study demonstrates that (a) providing code‑like intermediate representations dramatically improves LLM performance on logic‑dense extraction tasks, and (b) the intrinsic reasoning capacity of the model is a critical factor for handling non‑linear rule dependencies. BREX fills a crucial benchmark gap, enabling systematic evaluation of logic‑aware LLMs. The authors envision extending BREX to multilingual settings, integrating the extracted rule flows directly into BPMN or Drools engines, and exploring automated prompt optimization for even richer logical structures.

In summary, the paper delivers a high‑quality, cross‑domain benchmark (BREX) and a versatile, prompt‑centric framework (ExIde) that together push the frontier of automated business rule flow modeling with large language models, offering both academic insight and practical pathways for industry adoption.