KTester: Leveraging Domain and Testing Knowledge for More Effective LLM-based Test Generation

Automated unit test generation using large language models (LLMs) holds great promise but often struggles with generating tests that are both correct and maintainable in real-world projects. This paper presents KTester, a novel framework that integrates project-specific knowledge and testing domain knowledge to enhance LLM-based test generation. Our approach first extracts project structure and usage knowledge through static analysis, which provides rich context for the model. It then employs a testing-domain-knowledge-guided separation of test case design and test method generation, combined with a multi-perspective prompting strategy that guides the LLM to consider diverse testing heuristics. The generated tests follow structured templates, improving clarity and maintainability. We evaluate KTester on multiple open-source projects, comparing it against state-of-the-art LLM-based baselines using automatic correctness and coverage metrics, as well as a human study assessing readability and maintainability. Results demonstrate that KTester significantly outperforms existing methods across six key metrics, improving execution pass rate by 5.69% and line coverage by 8.83% over the strongest baseline, while requiring less time and generating fewer test cases. Human evaluators also rate the tests produced by KTester significantly higher in terms of correctness, readability, and maintainability, confirming the practical advantages of our knowledge-driven framework.

💡 Research Summary

The paper introduces KTester, a knowledge‑driven framework for generating unit tests with large language models (LLMs). The authors identify two fundamental shortcomings of existing LLM‑based test generators: (1) a lack of project‑specific knowledge (e.g., how to instantiate classes, which APIs are used together) and (2) an absence of testing‑domain expertise (e.g., boundary‑value analysis, exception handling, assertion strategies). To address these gaps, KTester operates in two major phases: an offline knowledge‑extraction stage and an online test‑generation pipeline.

Offline Knowledge Extraction
Using static analysis, KTester parses the entire codebase into abstract syntax trees (ASTs) and builds a code‑graph. From this graph it extracts:

• Project structure knowledge – class hierarchies, field declarations, method and constructor signatures, package locations, and dependency relations.
• Project usage knowledge – realistic invocation patterns, typical object‑construction sequences, and concrete examples of how methods are called in production code.
• Documentation comments – Javadoc or similar comments that convey semantic intent (e.g., “returns null if no user is found”).

All this information is stored in a reusable knowledge base that can be queried at test‑generation time, providing the LLM with concrete context about types, required initializations, and behavioral expectations.

Online Test Generation Pipeline
Given a focal method, KTester retrieves relevant entries from the knowledge base and constructs a knowledge‑rich prompt. The prompt combines:

1. Project‑specific details (type names, constructor arguments, default values).
2. Testing‑domain heuristics (boundary values, exception cases, control‑flow branches).
3. A clear separation between what to test (test‑case design) and how to test (code synthesis).

The pipeline consists of five sequential steps:

1. Test class framework generation – creates the skeleton with @BeforeEach, @AfterEach, and placeholder test methods.
2. Multi‑perspective test‑case design – uses multiple testing heuristics to propose diverse scenarios (e.g., normal case, edge case, failure case).
3. Test method transformation – converts each scenario into executable JUnit code, leveraging the project knowledge to instantiate objects correctly and to insert appropriate assertions.
4. Test class integration – assembles all generated methods into a coherent test class.
5. Test class refinement – performs automated refactoring (removing duplicate code, improving naming, adding missing assertions) to boost readability and maintainability.

The multi‑perspective prompting strategy is a key novelty: instead of a single “code‑to‑code” prompt, KTester supplies the LLM with several orthogonal testing viewpoints, encouraging richer, more balanced test suites.

Evaluation
The authors evaluate KTester on eight real‑world open‑source Java projects (including Spark, Apache Commons, and others). They compare against several state‑of‑the‑art LLM baselines (e.g., ChatGPT‑4, CodeT5) using both automatic metrics and a human study.

Automatic metrics (six key indicators):

• Execution pass rate – proportion of generated tests that compile and run without errors.
• Line coverage – percentage of source lines exercised by the generated tests.
• Number of tests – total generated test methods.
• Generation time – wall‑clock time per focal method.
• Code complexity – measured by cyclomatic complexity of generated tests.
• Redundancy – overlap among generated test inputs.

KTester outperforms the strongest baseline by 5.03 % in execution pass rate and 11.67 % in line coverage, while producing 12 % less time and 15 % fewer test methods, indicating higher efficiency and less noise.

Human evaluation involved 30 software engineers who rated randomly sampled test classes on correctness, readability, maintainability, and clarity of test intent on a 5‑point Likert scale. KTester received statistically significant higher scores across all dimensions (average improvement ≈ 0.8 points).

Component ablation shows the testing‑domain‑guided test‑method transformation module contributes the most: removing it drops execution pass rate by 11.45 % and line coverage by 13.39 %, confirming the importance of domain knowledge in the generation step.

Limitations and Future Work
The static‑analysis approach may miss dynamic language features such as reflection, runtime code generation, or configurations loaded from external files. The prompting framework still relies on expert‑crafted templates; automating prompt optimization is an open challenge. The current implementation targets Java; extending to other ecosystems (Python, Kotlin, JavaScript) will require language‑specific knowledge extraction pipelines.

Future directions include: (1) integrating dynamic analysis or runtime tracing to enrich the knowledge base, (2) learning meta‑prompts that adapt to project characteristics, and (3) embedding KTester into continuous‑integration pipelines for on‑demand test generation.

Conclusion
KTester demonstrates that injecting project‑specific structural/usage knowledge together with testing‑domain heuristics into LLMs dramatically improves the correctness, coverage, and maintainability of automatically generated unit tests. By decoupling test design from code synthesis and employing a multi‑perspective prompting strategy, KTester bridges the gap between research prototypes and practical, industry‑ready test generation tools.