Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, a paradigm adopted by many high-ranking agents on the SWE-bench leaderboard. However, we observe that GPT-5.2, which writes almost no new tests, can even achieve performance comparable to top-ranking agents. This raises the critical question: whether such tests meaningfully improve issue resolution or merely mimic human testing practices while consuming a substantial interaction budget. To reveal the impact of agent-written tests, we present an empirical study that analyzes agent trajectories across six state-of-the-art LLMs on SWE-bench Verified. Our results show that while test writing is commonly adopted, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies Furthermore, these tests typically serve as observational feedback channels, where agents prefer value-revealing print statements significantly more than formal assertion-based checks. Based on these insights, we perform a controlled experiment by revising the prompts of four agents to either increase or reduce test writing. The results suggest that changes in the volume of agent-written tests do not significantly change final outcomes. Taken together, our study reveals that current test-writing practices may provide marginal utility in autonomous software engineering tasks.

💡 Research Summary

This paper investigates the practical value of tests generated on‑the‑fly by large language model (LLM) based code agents when they attempt to resolve real‑world GitHub issues. Using the SWE‑bench Verified benchmark (500 curated issues) and a lightweight “mini‑SWE‑agent” framework that only offers a Bash interface, the authors isolate the native testing behavior of six state‑of‑the‑art LLM families: Claude‑Opus‑4.5, Gemini‑3‑Pro‑Preview, GPT‑5.2, Kimi‑K2‑Thinking, Minimax‑M2, and DeepSeek‑V3.2‑Reasoner. By extracting file‑creation commands from the agents’ interaction logs, they identify test files that follow common Python naming conventions (e.g., test_*.py).

The study is organized around three research questions. RQ1 characterizes emergent testing behaviors—frequency, timing, and execution intensity. Results show that test writing is highly model‑dependent: Claude‑Opus‑4.5 creates at least one test in 83 % of tasks, while GPT‑5.2 does so in only 0.6 % of tasks, yet their overall issue‑resolution rates differ by a modest 2.6 percentage points (74.4 % vs. 71.8 %). Moreover, resolved and unresolved trajectories within the same model exhibit similar test‑writing frequencies; unsuccessful runs tend to spread test creation over more steps and execute tests more often, inflating API calls and token consumption without improving outcomes.

RQ2 examines the feedback signals encoded in the generated tests. By parsing test execution output, the authors find that value‑revealing print statements dominate (≈78 % of all output), while explicit assertions constitute only about 22 %. A rule‑based AST classifier groups assertions into four categories: exact‑value matches, local‑property checks, exception expectations, and relational/range constraints. The first two categories account for roughly 80 % of assertions, whereas relational checks are rare (<5 %). This indicates that agents primarily use tests as an observational channel rather than a rigorous specification oracle.

RQ3 evaluates causal impact by intervening on the prompt to either encourage or suppress test creation. When prompts explicitly ask agents to write and run new tests, overall success improves by roughly 2 % but at the cost of a 12–14 % increase in API calls and token usage. Conversely, prompting agents to avoid writing tests reduces success by about 3 % while cutting interaction costs by more than 15 %. Thus, the quantity of agent‑generated tests has only a marginal effect on final resolution rates, whereas the efficiency penalty can be substantial.

The paper’s contributions are threefold: (1) a comprehensive behavioral profiling of test generation across diverse LLM agents, revealing that test writing is a model‑specific style rather than a universal success driver; (2) a detailed analysis of the informational content of agent‑written tests, showing a strong bias toward simple prints and limited assertion diversity; (3) an empirical causality study demonstrating that large swings in test‑writing behavior translate into only modest changes in success but significant efficiency differences.

In conclusion, the authors argue that the prevalent practice of on‑the‑fly test generation in autonomous software‑engineering agents offers limited utility for bug fixing and may waste valuable interaction budget. Future work should explore more cost‑effective feedback mechanisms—such as tighter integration of static analysis, smarter assertion synthesis, or meta‑learning approaches that can infer high‑value checks without incurring the overhead of full test suites. This research opens a path toward designing LLM agents that balance correctness guarantees with practical resource constraints.