Coverage Isn't Enough: SBFL-Driven Insights into Manually Created vs. Automatically Generated Tests

The testing phase is an essential part of software development, but manually creating test cases can be time-consuming. Consequently, there is a growing need for more efficient testing methods. To reduce the burden on developers, various automated test generation tools have been developed, and several studies have been conducted to evaluate the effectiveness of the tests they produce. However, most of these studies focus primarily on coverage metrics, and only a few examine how well the tests support fault localization-particularly using artificial faults introduced through mutation testing. In this study, we compare the SBFL (Spectrum-Based Fault Localization) score and code coverage of automatically generated tests with those of manually created tests. The SBFL score indicates how accurately faults can be localized using SBFL techniques. By employing SBFL score as an evaluation metric-an approach rarely used in prior studies on test generation-we aim to provide new insights into the respective strengths and weaknesses of manually created and automatically generated tests. Our experimental results show that automatically generated tests achieve higher branch coverage than manually created tests, but their SBFL score is lower, especially for code with deeply nested structures. These findings offer guidance on how to effectively combine automatically generated and manually created testing approaches.

💡 Research Summary

The paper investigates the relative strengths and weaknesses of manually written unit tests (MC‑tests) and automatically generated unit tests (AG‑tests) produced by EvoSuite, using a novel evaluation metric: the Spectrum‑Based Fault Localization (SBFL) score. While many prior studies have compared test generation tools solely on coverage or mutation scores, this work adds a fault‑localization perspective, which more directly reflects a test suite’s usefulness for debugging real defects.

Research Context and Methodology
The authors selected the Defects4J benchmark (version 3.0.1), focusing on the Lang and Math projects, which together provide 167 buggy/fixed program versions. For each bug, they generated AG‑tests by running EvoSuite (v1.2.0) on the fixed version of the class containing the defect. This ensures that the generated tests can exercise the intended correct behavior and reach all reachable code paths. MC‑tests were taken directly from the developer‑written test suites that accompany each Defects4J entry.

Two evaluation dimensions were measured: (1) code coverage (statement and branch) using JaCoCo, and (2) the SBFL score, which quantifies how well a test suite enables fault localization. The SBFL score is computed as follows: (a) generate mutants with Mutanerator (13,444 mutants across all subjects, using 12 mutation operators covering conditionals, arithmetic, increments, negations, etc.); (b) run each mutant with the test suite and collect pass/fail outcomes and execution spectra; (c) calculate suspiciousness for each statement using the Ochiai formula; (d) rank statements by suspiciousness, normalize the rank to a 0‑1 scale, and assign the normalized rank of the mutated statement as the mutant’s rScore; (e) average rScore over all mutants to obtain the SBFL score for a given (test suite, mutant generator, program) triple.

Statistical analysis employed the Wilcoxon signed‑rank test to assess significance between MC‑tests and AG‑tests for both metrics. Additionally, the authors performed a depth‑of‑nesting analysis, grouping statements by the nesting level of their enclosing control structures, to explore how code complexity influences SBFL performance.

Key Findings

1. Coverage Advantage of AG‑tests – AG‑tests achieved higher branch coverage (≈9 % absolute improvement) and modestly higher statement coverage (≈4 % improvement) compared with MC‑tests. The average number of test cases was comparable (AG‑tests ≈ 112, MC‑tests ≈ 103), indicating that EvoSuite can produce a dense set of tests without excessive bloat.

2. SBFL Score Deficit of AG‑tests – Despite superior coverage, AG‑tests yielded a lower overall SBFL score (mean ≈ 0.42) than MC‑tests (mean ≈ 0.55). The gap widened dramatically for code with deeper nesting: for methods whose maximum nesting depth ≥ 3, the AG‑test SBFL score fell below 0.30, whereas MC‑tests remained above 0.45.

3. Mutation‑Operator Sensitivity – Conditional mutations (e.g., changing “<” to “≤”) produced the largest SBFL score disparity, with AG‑tests performing poorly. Arithmetic mutations showed a smaller gap, suggesting that EvoSuite’s search heuristics are better at exercising arithmetic paths than complex logical branches.

4. Statistical Significance – Both coverage and SBFL differences were statistically significant (p < 0.01) under the Wilcoxon test, confirming that the observed trends are not due to random variation.

Interpretation and Implications
The results demonstrate that high coverage does not guarantee effective fault localization. AG‑tests excel at exercising many branches, especially simple conditional structures, but they often lack targeted failing test cases that highlight the precise statements responsible for a defect. Consequently, the suspiciousness scores derived from AG‑tests are diluted, leading to lower SBFL rankings. In contrast, manually crafted tests, informed by developer intuition and domain knowledge, tend to include boundary‑value and error‑condition inputs that generate informative failures, thereby boosting SBFL effectiveness.

The authors argue that a hybrid testing strategy is advisable: use automated generation to quickly achieve broad coverage and uncover shallow bugs, then augment the suite with manually designed tests for high‑risk, deeply nested modules where fault localization is critical.

Threats to Validity
A notable limitation is the use of the fixed version of the program for test generation. In practice, developers write tests against buggy code, and the dynamics of test evolution may affect both coverage and SBFL outcomes. Moreover, the study focuses solely on EvoSuite; results may differ with other generation tools (e.g., Randoop, Agitar). Parameter settings for EvoSuite were left at defaults, which could influence the balance between test diversity and redundancy. Finally, the SBFL score relies on synthetic mutants; while mutation testing is a widely accepted proxy for real faults, the correlation with actual bug localization performance warrants further empirical validation.

Conclusion
The paper makes a compelling case that “coverage isn’t enough.” By introducing the SBFL score as an evaluation metric, it reveals a nuanced trade‑off: automated test generation provides superior branch coverage but falls short in supporting effective fault localization, especially for complex, deeply nested code. The findings encourage practitioners to combine automated and manual testing approaches, leveraging the strengths of each to build more robust and debuggable test suites. Future work could explore adaptive generation techniques that explicitly aim to produce informative failing tests, evaluate additional tools, and validate SBFL‑based findings against real-world bug reports.