Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering

Large language models for code are advancing fast, yet our ability to evaluate them lags behind. Current benchmarks focus on narrow tasks and single metrics, which hide critical gaps in robustness, interpretability, fairness, efficiency, and real-world usability. They also suffer from inconsistent data engineering practices, limited software engineering context, and widespread contamination issues. To understand these problems and chart a path forward, we combined an in-depth survey of existing benchmarks with insights gathered from a dedicated community workshop. We identified three core barriers to reliable evaluation: the absence of software-engineering-rich datasets, overreliance on ML-centric metrics, and the lack of standardized, reproducible data pipelines. Building on these findings, we introduce BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation. BEHELM provides a structured way to assess models across tasks, languages, input and output granularities, and key quality dimensions. Our goal is to reduce the overhead currently required to construct benchmarks while enabling a fair, realistic, and future-proof assessment of LLMs in software engineering.

💡 Research Summary

The paper “Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering” addresses a critical gap in the evaluation of large language models (LLMs) that generate or manipulate code. While LLMs such as GitHub Copilot, Google CodeBot, and emerging instruction‑tuned models have demonstrated impressive capabilities across tasks like code completion, summarization, bug fixing, test generation, and multi‑modal agent assistance, the authors argue that current benchmarking practices are insufficient for measuring real‑world usefulness.

Through an extensive literature survey of existing SE‑focused benchmarks (HumanEval, MBPP, APPS, LiveCodeBench, ClassEval, EvoCodeBench, SWE‑bench and its variants, CVE‑xes, SecBench, TestGenEval, CodeXGLUE, etc.) and a dedicated community workshop at FORGE ‘26, the authors identify three systemic barriers:

1. Lack of software‑engineering‑rich datasets – most datasets contain only isolated code snippets, input‑output pairs, or natural‑language prompts, ignoring essential artifacts such as repository structure, build configurations, dependency manifests, commit histories, code‑review comments, issue discussions, and architectural constraints. Consequently, models are evaluated on syntactic correctness without assessing integration, maintainability, or adherence to project conventions.

2. Over‑reliance on ML‑centric metrics – prevailing metrics (accuracy, precision, recall, F1, BLEU, CodeBLEU, Pass@k, BERTScore) capture binary success or surface similarity but fail to reflect partial correctness, semantic equivalence, efficiency, fairness, robustness to distribution shift, interpretability, security, or maintainability. The “SWE‑bench Illusion” studies cited show that high scores can be achieved through memorization or n‑gram overlap rather than genuine reasoning.

3. Absence of standardized, reproducible data pipelines – each research group builds its own benchmark from scratch, leading to heterogeneous preprocessing, undocumented provenance, duplicated effort, and high risk of data contamination (code reuse across projects, leakage from training corpora). The paper notes that up to 75 % of a paper’s effort can be spent on dataset construction, stifling innovation.

To overcome these obstacles, the authors propose BEHELM (Benchmarking Infrastructure for Holistic Evaluation of LLMs in Software Engineering). BEHELM is a modular, community‑driven platform that unifies three pillars:

• Software‑scenario specification – a schema that captures the full development context (project layout, build scripts, dependency files, version‑control metadata, issue/PR discussions). This enables class‑level or repository‑level evaluation rather than isolated function‑level tests.

• Multi‑metric evaluation framework – beyond traditional accuracy, BEHELM incorporates metrics for interpretability (e.g., causal attribution of generated tokens), efficiency (GPU hours, latency), fairness/bias (language‑specific or framework‑specific disparities), robustness (performance under adversarial inputs or distribution shift), maintainability (cyclomatic complexity, code smell detection), and security (vulnerability patterns, unsafe API usage).

• Standardized data engineering pipelines – automated workflows for data collection, cleaning, deduplication, provenance tracking, contamination detection, expert annotation, and versioned releases. The pipeline logs every transformation, enabling full reproducibility and facilitating continuous benchmark updates.

A distinctive feature of BEHELM is milestone‑based evaluation for agentic systems. Instead of a single pass/fail outcome, the framework records progress across development stages (code synthesis, review, test creation, integration, deployment). This process‑aware assessment aligns with how developers actually interact with AI assistants and supports fine‑grained analysis of partial successes.

The paper also introduces an RL‑driven data generation component that synthesizes new, diverse coding tasks while automatically checking for overlap with existing training data, thereby mitigating contamination.

Empirical observations using BEHELM reveal stark performance gaps: models that achieve >90 % on HumanEval drop to ~38 % on class‑level tasks that require correct imports, build configuration, and adherence to project conventions. Similarly, bug‑fixing models that score >70 % on SWE‑bench verified fall to ~23 % on the commercial‑code SWE‑bench Pro set, underscoring memorization effects.

In conclusion, the authors argue that BEHELM provides the necessary infrastructure to transition from narrow, accuracy‑centric benchmarks to a holistic, reproducible, and real‑world‑oriented evaluation ecosystem for code‑focused LLMs. By standardizing datasets, expanding metric suites, and automating pipelines, BEHELM aims to reduce redundant engineering effort, foster community collaboration, and ultimately guide the development of trustworthy, efficient, and developer‑friendly AI coding assistants.