OmniCode: A Benchmark for Evaluating Software Engineering Agents

LLM-powered coding agents are redefining how real-world software is developed. To drive the research towards better coding agents, we require challenging benchmarks that can rigorously evaluate the ability of such agents to perform various software engineering tasks. However, popular coding benchmarks such as HumanEval and SWE-Bench focus on narrowly scoped tasks such as competition programming and patch generation. In reality, software engineers have to handle a broader set of tasks for real-world software development. To address this gap, we propose OmniCode, a novel software engineering benchmark that contains a broader and more diverse set of task categories beyond code or patch generation. Overall, OmniCode contains 1794 tasks spanning three programming languages (Python, Java, and C++) and four key categories: bug fixing, test generation, code review fixing, and style fixing. In contrast to prior software engineering benchmarks, the tasks in OmniCode are (1) manually validated to eliminate ill-defined problems, and (2) synthetically crafted or recently curated to avoid data leakage issues, presenting a new framework for synthetically generating diverse software tasks from limited real-world data. We evaluate OmniCode with popular agent frameworks such as SWE-Agent and show that while they may perform well on bug fixing for Python, they fall short on tasks such as Test Generation and in languages such as C++ and Java. For instance, SWE-Agent achieves a maximum of 20.9% with DeepSeek-V3.1 on Java Test Generation tasks. OmniCode aims to serve as a robust benchmark and spur the development of agents that can perform well across different aspects of software development. Code and data are available at https://github.com/seal-research/OmniCode.

💡 Research Summary

OmniCode is introduced as a comprehensive benchmark for evaluating large‑language‑model (LLM) coding agents across the full software development lifecycle. Unlike prior benchmarks such as HumanEval or SWE‑Bench that focus on isolated tasks like competition programming or single‑issue bug fixing, OmniCode comprises 1,794 tasks spanning three popular languages—Python, Java, and C++—and four distinct categories: bug fixing, test generation, code‑review response, and style fixing. The authors first collect 494 real‑world pull requests from 27 diverse open‑source repositories, ensuring each instance includes an issue description, a test suite, and a gold‑standard patch. They then apply language‑specific augmentation pipelines to synthesize additional data: multiple plausible but incorrect “bad patches” are generated using a mix of LLMs, review comments are created by prompting LLMs with bad patches and the correct solution, and style violations are injected via static analysis tools. All tasks undergo manual validation to eliminate ill‑defined cases.

The benchmark is used to evaluate two prominent agent frameworks, SWE‑Agent and Aider, each combined with four LLM back‑ends (Gemini 2.5 Flash, DeepSeek‑V3.1, GPT‑5‑mini, Qwen3‑32B). Results show that while agents achieve moderate success on Python bug‑fixing (≈60 % pass rate), performance drops sharply for test generation (maximum 20.9 % on Java), code‑review response (≈52 % on Python, <30 % on Java/C++), and style fixing (good on Python, poor on Java/C++). Aider consistently underperforms SWE‑Agent, especially on C++ tasks.

Key contributions include (1) a multi‑task, multi‑language benchmark that mirrors real development workflows, (2) a reproducible pipeline for generating synthetic but realistic test cases, bad patches, and review feedback from limited real data, and (3) a rigorous evaluation protocol that requires generated tests to pass the gold patch while failing all bad patches, thereby reducing trivial solutions. The paper also discusses limitations such as the representativeness of synthetic bad patches, the coverage of style‑checking tools across projects, and the current language scope. Future work is suggested to expand language coverage, incorporate interactive human‑LLM collaboration scenarios, and extend metrics to security and performance aspects. OmniCode thus provides a robust platform to drive the next generation of coding agents toward truly end‑to‑end software development capabilities.