SecCodeBench-V2 Technical Report

We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots’ capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group’s industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.

💡 Research Summary

SecCodeBench‑V2 is a publicly released benchmark designed to rigorously evaluate the secure code generation and repair capabilities of large language model (LLM) coding assistants. The benchmark consists of 98 scenarios derived from de‑identified internal vulnerability cases at Alibaba Group, covering five programming languages (Java, C, Python, Go, JavaScript) and 22 CWE categories. Each scenario follows a function‑level formulation: a complete project scaffold is provided, and the model must either implement a missing target function (generation) or patch a vulnerable function (repair) under fixed interfaces and dependency constraints.

Four prompt variants are supported for each scenario: plain generation (gen), generation with an explicit safety hint (gen‑hints), plain repair (fix), and repair with a safety hint (fix‑hints). This design captures a spectrum of realistic developer workflows, from writing new code to fixing existing insecure code, with or without guidance.

The benchmark’s core contribution lies in its evaluation pipeline. For every scenario, a Docker‑isolated runtime environment is prepared. The pipeline proceeds in five stages: (1) initialization of configuration, test suites, and LLM API availability; (2) prompting the LLM with the appropriate scenario‑specific prompt and, if needed, iterative error‑driven repair; (3) batch execution where language‑specific validators compile and run the generated artifact, first checking functional correctness and then executing dedicated proof‑of‑concept (PoC) tests that attempt to trigger the vulnerability; (4) result analysis that aggregates multi‑round outcomes, computes Pass@K (default K = 1), and applies severity‑aware weighting across CWE severity levels (medium, high, critical) and scenario difficulty; (5) output generation, producing a leaderboard, detailed logs, and reproducibility artifacts.

Dynamic execution is the primary security oracle. Functional tests must pass before any security assessment, mirroring real‑world development where a program must first be runnable. PoC tests are crafted by security experts to reliably expose the specific vulnerability (e.g., memory‑safety violation, SQL injection, command injection). For vulnerability types where deterministic tests are insufficient—such as weak cryptographic choices, hard‑coded credentials, or information leakage—the pipeline falls back to an LLM‑as‑a‑judge mechanism. Multiple LLM judges independently evaluate the code’s security, and a majority‑vote determines the final verdict, with all judgments logged for transparency.

SecCodeBench‑V2 addresses three major shortcomings of prior secure‑coding benchmarks: (1) data contamination and lack of realism, by sourcing cases from internal, non‑public codebases and providing full project contexts; (2) reliance on static analysis, by employing execution‑driven validation that can capture runtime‑only exploits; (3) coarse scoring, by introducing a Pass@K metric enriched with CWE severity weighting and scenario‑level aggregation, yielding a nuanced view of model performance across functional and security dimensions.

The framework is modular and extensible: new languages, validation back‑ends, or LLM providers can be added with minimal effort. Security is enforced through container isolation, preventing generated code from affecting the host system. Fairness is promoted by multi‑round evaluation, deterministic seeding, and majority‑vote LLM judging. Usability is enhanced by configuration‑file driven parameters and standardized logging, facilitating debugging and reproducibility.

In summary, SecCodeBench‑V2 provides a rigorous, reproducible, and industry‑relevant foundation for assessing the security posture of AI coding assistants. By combining realistic, function‑level tasks, dynamic execution‑based security testing, and a sophisticated scoring scheme, it enables researchers and practitioners to benchmark, compare, and improve LLMs’ ability to generate and repair secure code across diverse languages and vulnerability types. All artifacts, including test cases, Docker images, and evaluation scripts, are publicly available at the provided GitHub and project website.