Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects

In the literature, there is a rather clear segregation between manually written tests by developers and automatically generated ones. In this paper, we explore a third solution: to automatically improve existing test cases written by developers. We present the concept, design, and implementation of a system called \dspot, that takes developer-written test cases as input (junit tests in Java) and synthesizes improved versions of them as output. Those test improvements are given back to developers as patches or pull requests, that can be directly integrated in the main branch of the test code base. We have evaluated DSpot in a deep, systematic manner over 40 real-world unit test classes from 10 notable and open-source software projects. We have amplified all test methods from those 40 unit test classes. In 26/40 cases, DSpot is able to automatically improve the test under study, by triggering new behaviors and adding new valuable assertions. Next, for ten projects under consideration, we have proposed a test improvement automatically synthesized by \dspot to the lead developers. In total, 13/19 proposed test improvements were accepted by the developers and merged into the main code base. This shows that DSpot is capable of automatically improving unit-tests in real-world, large-scale Java software.

💡 Research Summary

The paper introduces a novel research direction called Automatic Test Improvement (ATI), which sits between manually written unit tests and fully automatically generated tests. The authors present DSpot, a tool that takes existing JUnit test cases as input, automatically modifies them, and outputs improved versions that can be submitted to developers as patches or pull requests. DSpot combines two well‑known techniques: evolutionary input amplification (based on Tonella’s work) and regression oracle generation (based on Xie’s work).

In the input amplification phase, DSpot systematically mutates literals, method calls, and object constructions within the original test. Numeric literals are altered using +1, –1, ×2, ÷2, or replaced by another literal of the same type. Strings are mutated by inserting, deleting, or replacing characters, or by generating a random string of equal length. Booleans are simply negated. Method calls can be duplicated, removed, or new calls added using existing variables as targets. When a new object or primitive argument is required, DSpot creates it using a default constructor or a randomly generated value. These transformations are applied iteratively, each iteration building on the tests generated in the previous one, thereby exploring a large input space.

The assertion amplification phase instruments the test to capture the state of all objects after the execution of the test body but before the original assertions. It does this by inserting observation points that invoke all getter methods. After running the instrumented test, DSpot records the concrete values returned by these getters and generates new JUnit assertions that compare the observed values to the recorded ones. If the mutated input triggers an exception, DSpot also adds an assertion that the specific exception is thrown.

Both phases are guided by the mutation score, a metric that measures how many artificial faults (mutants) a test suite can detect. DSpot evaluates the mutation score of each candidate test variant and retains only those that improve the score. This design choice aligns the tool’s objective with developers’ intuition that a higher mutation score correlates with higher fault‑detection capability.

The empirical evaluation involved ten mature open‑source Java projects, from which 40 test classes (approximately 200 test methods) were selected. DSpot amplified every test method in these classes. In 26 out of 40 cases, the tool produced a variant with a higher mutation score than the original. Notably, DSpot succeeded in improving a test class that already had a 99 % mutation score, raising it to 100 % by adding a single assertion. The time required for amplification was typically a few minutes per class, demonstrating practical feasibility.

To assess real‑world relevance, the authors submitted 19 pull requests containing DSpot‑generated improvements to the maintainers of the ten projects. Thirteen of these (≈68 %) were accepted and merged into the main branches. Accepted patches mainly introduced new assertions that increased coverage of previously untested branches or validated state changes introduced by the input amplification. Interviews with developers revealed that the automatically generated patches were easy to understand, did not degrade code readability, and saved the effort of manually writing additional assertions.

The paper discusses several threats to validity. Relying solely on mutation score may overlook other quality aspects such as test readability or execution performance. The computational cost of generating and evaluating many test variants could become prohibitive for very large test suites. Moreover, DSpot currently supports only JUnit tests for Java; extending the approach to other languages, testing frameworks, or property‑based testing would require additional research.

In summary, DSpot demonstrates that automatic improvement of existing developer‑written tests is both technically feasible and practically valuable. By combining systematic input space exploration with automated oracle generation, and by using mutation score as a guiding metric, DSpot can produce test enhancements that developers are willing to adopt. The authors have released both the tool and the experimental data as open‑source, inviting further replication and extension by the research community.