ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design

While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs’ performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, and reference model generation. Our benchmark features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases, and 132 reference model samples across Python, SystemC, and CXXRTL. Evaluation results reveal substantial performance gaps, with state-of-the-art Claude-4.5-opus achieving only 30.74% on Verilog generation and 13.33% on Python reference model generation, demonstrating significant challenges compared to existing saturated benchmarks where SOTA models achieve over 95% pass rates. Additionally, to help enhance LLM reference model generation, we provide an automated toolbox for high-quality training data generation, facilitating future research in this underexplored domain. Our code is available at https://github.com/zhongkaiyu/ChipBench.git.

💡 Research Summary

The paper “ChipBench: A Next‑Step Benchmark for Evaluating LLM Performance in AI‑Aided Chip Design” addresses a critical gap in the evaluation of large language models (LLMs) for hardware engineering. Existing benchmarks such as VerilogEval and RTLLM have become saturated; state‑of‑the‑art LLM‑based Verilog generators achieve >95 % pass rates on these tests, yet the tasks are far too simplistic compared to real‑world chip design. They typically involve short, self‑contained modules (10–76 lines) that never instantiate sub‑modules, and they are sourced from coding contests rather than industrial code bases. Consequently, high benchmark scores do not guarantee that an LLM will be useful in a production environment where modules often exceed 10 000 lines, contain hierarchical designs, and must interoperate with many other IP blocks.

To remedy this, the authors propose ChipBench, a comprehensive benchmark that expands evaluation beyond pure code generation to include debugging and reference‑model generation—two activities that are essential in industry but have been largely ignored in prior work. ChipBench comprises three task families:

1. Verilog Generation – 44 realistic modules drawn from open‑source CPU IPs and competitive platforms. The modules are categorized as self‑contained, non‑self‑contained (hierarchical), and CPU‑IP sub‑modules. Compared with VerilogEval, the average code length is 3.8× larger (≈61.7 lines vs. 16.1) and the average cell count is 13.9× larger (≈438.7 vs. 31.4). For each case the benchmark provides a manually crafted description prompt, a golden Verilog implementation, and a robust testbench that mixes hand‑designed corner cases with >1 000 random stimuli. Evaluation proceeds by syntax checking the generated file, then running iVerilog to compare functional outputs against the golden reference.

2. Verilog Debugging – 89 injected‑bug cases covering four error classes common in hardware design: arithmetic, assignment, timing, and state‑machine bugs. Bugs are introduced manually into the golden modules, and two evaluation modes are offered. In the zero‑shot mode the model receives only the buggy code and a statement that an error exists; in the one‑shot mode it also receives a VCD waveform file, mimicking the way verification engineers use simulation traces to locate faults. This design tests LLMs’ ability to understand, localize, and repair hardware bugs—an ability that is arguably more valuable for immediate industrial adoption than raw code generation.

3. Reference Model Generation – 132 samples (44 modules × 3 target languages) requiring the model to produce high‑level behavioral models in Python, SystemC, or CXXRTL. The benchmark includes a custom Heterogeneous Test Engine (HTE) that automatically extracts I/O ports from the golden Verilog, compiles the Verilog to C++ via Verilator, and generates a C++ test harness. The LLM‑generated reference model is first syntax‑checked, then functionally verified by comparing its outputs against the compiled golden model across a suite of stimuli.

The authors evaluate several leading LLMs, with Claude‑4.5‑opus (Anthropic, 2025) serving as the primary reference. Results reveal a stark performance gap: on Verilog generation the model achieves only 30.74 % pass rate, far below the >95 % reported on older benchmarks. On Python reference‑model generation the pass rate drops further to 13.33 %. Debugging performance is modestly better, with 5–20 % higher pass rates than generation, but still insufficient for production use. Notably, the hierarchical (non‑self‑contained) and CPU‑IP categories prove especially challenging, with most models failing to produce correct top‑level code when sub‑module definitions are provided.

Beyond the benchmark itself, the paper contributes an automated toolbox for generating high‑quality reference‑model training data. Using the toolbox, the authors transformed 10 000 Verilog samples from the QiMeng CodeV‑R1 dataset into 2 206 validated Python reference models, demonstrating scalability to larger corpora such as Pyranet (≈692 k samples) and VeriGen (≈50 k samples). This addresses a current bottleneck: the scarcity of paired Verilog‑to‑high‑level‑model data for supervised LLM training.

In summary, ChipBench offers a realistic, multi‑dimensional evaluation suite that mirrors the full chip‑design workflow: (i) writing RTL, (ii) locating and fixing bugs, and (iii) producing high‑level reference models for verification. The benchmark exposes substantial deficiencies in current LLMs, indicating that despite impressive results on narrow code‑generation tasks, they are far from ready for industrial chip design. The released dataset, evaluation framework, and data‑generation toolbox provide a solid foundation for future research aimed at closing this gap, encouraging the community to develop models that understand hierarchical hardware structures, perform reliable debugging, and generate accurate behavioral models.