QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design

Quantum computing is an emerging field recognized for the significant speedup it offers over classical computing through quantum algorithms. However, designing and implementing quantum algorithms pose challenges due to the complex nature of quantum mechanics and the necessity for precise control over quantum states. Despite the significant advancements in AI, there has been a lack of datasets specifically tailored for this purpose. In this work, we introduce QCircuitBench, the first benchmark dataset designed to evaluate AI’s capability in designing and implementing quantum algorithms using quantum programming languages. Unlike using AI for writing traditional codes, this task is fundamentally more complicated due to highly flexible design space. Our key contributions include: 1. A general framework which formulates the key features of quantum algorithm design for Large Language Models. 2. Implementations for quantum algorithms from basic primitives to advanced applications, spanning 3 task suites, 25 algorithms, and 120,290 data points. 3. Automatic validation and verification functions, allowing for iterative evaluation and interactive reasoning without human inspection. 4. Promising potential as a training dataset through preliminary fine-tuning results. We observed several interesting experimental phenomena: LLMs tend to exhibit consistent error patterns, and fine-tuning does not always outperform few-shot learning. In all, QCircuitBench is a comprehensive benchmark for LLM-driven quantum algorithm design, and it reveals limitations of LLMs in this domain.

💡 Research Summary

QCircuitBench is introduced as the first large‑scale benchmark dataset specifically designed to evaluate large language models (LLMs) on the task of quantum algorithm design and implementation. The authors argue that, unlike conventional code‑generation benchmarks where abundant textual data exist, quantum algorithm design suffers from a scarcity of structured examples, a highly flexible design space, and the need for precise mathematical and physical reasoning. To address these challenges, the paper proposes a comprehensive framework that treats quantum algorithm design as a code‑generation problem, thereby enabling exact, machine‑readable representations and automated verification. The dataset comprises three distinct task suites: (1) Oracle Construction, (2) Quantum Algorithm Design, and (3) Random Circuit Synthesis. Across these suites, 25 representative algorithms are covered—including textbook examples (Bernstein‑Vazirani, Deutsch‑Jozsa, Simon, Grover, Phase Estimation, QFT, GHZ/W‑state preparation), advanced problems (Generalized Simon’s problem, Shor’s factoring), variational quantum algorithms (VQE, QAOA, QAE, ENC), and quantum communication protocols (teleportation, QKD). In total, 120,290 data points are provided, with 35,872 for oracle construction, 6,534 for algorithm design, and 77,884 for random circuit synthesis. Each data point contains seven tightly coupled components: a natural‑language problem description (augmented with LaTeX formulas), generation code (Qiskit), the target OpenQASM 3.0 circuit, optional custom‑gate definitions, a classical post‑processing function (Python), an explicit oracle/gate definition file, and an automated verification suite (unit tests). QCircuitBench is released for both Qiskit + OpenQASM and Cirq, ensuring platform‑agnostic usability. Automatic verification checks syntax, functional correctness, and query‑complexity specifications (e.g., shot count), allowing human‑free, large‑scale evaluation. Preliminary experiments with state‑of‑the‑art LLMs (GPT‑4, Claude, LLaMA) under few‑shot and fine‑tuning regimes reveal consistent error patterns: incorrect oracle I/O mapping, missing classical post‑processing, and misuse of custom multi‑controlled gates. Notably, fine‑tuning does not universally outperform few‑shot prompting, suggesting that mere data volume is insufficient for mastering the intricate constraints of quantum algorithm design. The authors compare QCircuitBench to existing quantum circuit benchmarks such as QASMBench, MQTBench, and VeriQBench, highlighting that those resources focus on hardware performance evaluation and lack oracle construction, classical post‑processing, and query‑complexity details essential for AI‑driven design. By providing a richly annotated, automatically verifiable dataset, QCircuitBench establishes a new standard for assessing and training LLMs in the quantum computing domain, and it opens avenues for future research on model architectures, prompting strategies, and domain‑specific fine‑tuning to bridge the gap between AI‑generated code and physically realizable quantum algorithms.