ProOPF: Benchmarking and Improving LLMs for Professional-Grade Power Systems Optimization Modeling

Growing renewable penetration introduces substantial uncertainty into power system operations, necessitating frequent adaptation of dispatch objectives and constraints and challenging expertise-intensive, near-real-time modeling workflows. Large Language Models (LLMs) provide a promising avenue for automating this process by translating natural-language (NL) operational requirements into executable optimization models via semantic reasoning and code synthesis. Yet existing LLM datasets and benchmarks for optimization modeling primarily target coarse-grained cross-domain generalization, offering limited, rigorous evaluation in power-system settings, particularly for Optimal Power Flow (OPF). We therefore introduce \textbf{ProOPF-D} and \textbf{ProOPF-B}, a dataset and benchmark for professional-grade OPF modeling: ProOPF-D contains 12K instances pairing NL requests with parameter adjustments and structural extensions to a canonical OPF, together with executable implementations; ProOPF-B provides 121 expert-annotated test cases with ground-truth code, enabling end-to-end evaluation under both concrete and abstract OPF modeling regimes.

💡 Research Summary

The paper addresses the growing need for rapid, expert‑level adaptation of Optimal Power Flow (OPF) models in power systems that are increasingly stressed by high renewable penetration and associated uncertainty. While large language models (LLMs) such as GPT‑4, Claude‑Sonnet, and DeepSeek‑3.2 have demonstrated impressive code‑generation capabilities, existing natural‑language‑to‑optimization benchmarks (e.g., NL4Opt, MAMO, OptiBench) focus on coarse‑grained, cross‑domain generalization and do not rigorously test the physical consistency required for power‑system problems. To fill this gap, the authors introduce two complementary contributions: the ProOPF‑D dataset and the ProOPF‑B benchmark.

ProOPF‑D (Dataset)
ProOPF‑D contains 12 000 synthetic instances organized into four difficulty levels. Each instance is a triple {P, M, I}: a natural‑language description (P) of the operational request, a model specification (M) that encodes the required modifications, and an executable implementation (I) in a standard programming language. The key innovation is a “modification‑based” representation: every instance is expressed as a set of parameter patches Δπ and optional structural operators s applied to a canonical base OPF model Q₀. Parameter patches are structured tuples (component type, target identifier, operation, value), while structural operators consist of three parts—problem type (sₚ), constraint extensions (s_c), and objective modifications (s_o). This design isolates the varying parts of an OPF problem, avoiding the brittle regeneration of the entire model and ensuring that all generated instances respect a curated set of admissible modifications (Ωπ, Ωs) that preserve power‑flow physics. Difficulty levels are defined by two binary axes: (i) whether the NL request explicitly lists the parameter changes or requires inference, and (ii) whether a structural extension beyond the base model is needed. The Cartesian product yields four levels, ranging from simple explicit parameter tweaks to full inference plus structural redesign.

ProOPF‑B (Benchmark)
ProOPF‑B selects 121 real‑world OPF variants from the literature, each annotated by domain experts with ground‑truth code and a detailed model description. The benchmark defines two evaluation modes. In the “concrete” mode, the generated code must match the reference implementation exactly (including solver settings). In the “abstract” mode, only the logical equivalence of the model—objective, constraints, and decision variables—needs to be verified, allowing flexibility in coding style while still demanding physical correctness.

Experimental Findings
The authors evaluate several state‑of‑the‑art LLMs on both existing benchmarks and ProOPF‑B. On traditional cross‑domain benchmarks, the models achieve 85–95 % accuracy, confirming their strong general code‑generation abilities. However, on ProOPF‑B the average accuracy drops below 30 %, with the hardest level (Level 4) falling to around 12 %. This stark performance gap highlights that current LLMs struggle to internalize the tightly coupled nonlinear constraints, thermal limits, and security considerations that characterize professional OPF modeling.

Related Work and Distinction
Prior benchmarks either focus on linear/mixed‑integer programs or generate problem instances without enforcing domain‑specific physics, leading to a scale mismatch: typical OPF problems involve dozens of variables and dozens of constraints, whereas many existing datasets feature fewer than 15. Moreover, most data‑generation pipelines either rewrite natural‑language problem statements (problem‑centric) or perturb existing models (model‑centric) without a systematic way to guarantee physical feasibility. ProOPF‑D uniquely combines a model‑centric approach with expert‑curated admissible modification sets, ensuring every instance is both mathematically solvable and physically valid.

Contributions

1. Introduction of the first power‑system‑specific dataset and benchmark for LLM‑based optimization modeling.
2. A novel multi‑level, modification‑based dataset construction pipeline that isolates parameter updates and structural extensions while preserving a canonical OPF backbone.
3. An expert‑validated benchmark with concrete and abstract evaluation protocols, enabling fine‑grained assessment of LLM competence in professional OPF tasks.
4. Empirical evidence that current LLMs, despite high performance on generic benchmarks, are far from ready for domain‑critical power‑system modeling.

Future Directions
The paper suggests several avenues to close the gap: (i) designing physics‑aware prompts or chain‑of‑thought reasoning that explicitly references power‑flow equations; (ii) pre‑training LLMs on domain‑specific corpora such as grid simulation data, SCADA logs, and historical OPF solutions; (iii) tokenizing structural operators to give LLMs clearer signals for constraint and objective modifications; and (iv) exploring multimodal inputs that combine graph representations of network topology with natural language.

In summary, ProOPF‑D and ProOPF‑B provide a rigorous, scalable framework for evaluating and improving LLMs in the highly specialized arena of power‑system optimization, exposing current limitations and charting a clear research roadmap toward trustworthy, expert‑grade AI assistance in grid operations.