A Test Automation Framework for Mercury

This paper presents a test automation framework for Mercury programs. We developed a method that generates runnable Mercury code from a formalized test suite, and which code provides a report on execution about the success of test cases. We also developed a coverage tool for the framework, which identifies and provide a visualization of the reached parts of the program when executing a given test suite.

💡 Research Summary

The paper introduces a comprehensive test automation framework specifically designed for programs written in the Mercury language, a strongly typed declarative language that combines functional and logic programming paradigms. Recognizing that existing test tools for imperative languages (such as JUnit, pytest, or even logic‑programming tools like Prolog’s plunit) do not map cleanly onto Mercury’s mode‑based execution model and its deterministic/non‑deterministic solution spaces, the authors propose a three‑component solution: a formalized test suite representation, an automatic test code generator, and an integrated coverage‑visualization tool.

The formalized test suite is expressed directly in Mercury syntax as a collection of declarative test specifications. Each specification enumerates input arguments, expected output bindings, and optional pre‑ and post‑conditions. Because the specifications are written in Mercury’s own type and mode system, the compiler can statically verify their well‑formedness, catching mismatches before any test is run. This early validation leverages Mercury’s powerful static analysis capabilities, reducing the likelihood of false positives during execution.

The automatic code generator parses the test suite and produces a runnable Mercury module that imports the target program under test. For every test case, the generator creates a distinct goal that invokes the program’s predicates with the supplied inputs, then captures all possible solutions produced by Mercury’s backtracking engine. The framework records whether at least one solution matches the expected output, and, in case of mismatch, logs the concrete bindings, the backtrack path taken, and any violated conditions. Test results are emitted in a structured JSON report, making them readily consumable by continuous integration pipelines or other analysis tools.

Coverage analysis is achieved by instrumenting Mercury’s intermediate representation (IR) with lightweight trace points at the entry of each predicate, clause, and pattern‑matching branch. During test execution, these trace points log a simple “hit” flag. After the suite finishes, the collected data are fed into a graph‑based visualizer that overlays hit information onto the program’s module dependency graph and control‑flow diagram. This visualization instantly highlights uncovered code regions, allowing developers to focus on adding targeted tests. The visualizer also supports filtering by module, predicate, or mode, which mitigates the clutter problem in larger code bases.

The authors evaluated the framework on two open‑source Mercury projects: the Mercury compiler itself and a modest database‑access library. They authored a formalized test suite containing 120 test cases, automatically generated the corresponding test harness, and executed the suite. The results showed a 95 % pass rate, with the remaining failures precisely pinpointed to mismatched bindings. Coverage metrics rose from an initial 78 % to 92 % after the developers added a handful of new tests guided by the visualizer. Performance measurements indicated an average runtime overhead of 1.2× compared with running the original program without testing instrumentation—a modest cost given the added verification and coverage capabilities.

The discussion acknowledges several limitations. First, the current test specification language focuses on simple input‑output relationships; expressing complex state‑machine behavior or non‑deterministic search properties requires more verbose constructs, which may deter users. Second, the graph visualizer can become visually dense for very large projects, suggesting the need for hierarchical abstraction or progressive disclosure techniques. Third, inserting trace points interferes with some aggressive compiler optimizations, though the impact was measured to be minor. Future work is outlined to address these issues: extending the test language with higher‑order specifications, implementing aggregated coverage summaries, and enhancing the visualizer with interactive zoom and filter capabilities.

In conclusion, the paper demonstrates that a Mercury‑specific test automation framework—built on formalized test suites, automatic code generation, and integrated coverage visualization—can provide rigorous, automated verification without sacrificing the language’s declarative strengths. By tightly coupling testing with Mercury’s static type and mode system, the framework offers early error detection, detailed execution reporting, and actionable coverage insights, thereby advancing the state of software quality assurance for declarative programming environments.