Mutation Sampling Technique for the Generation of Structural Test Data

The presented approach addresses the data generation problem for both validation and physical tests. Data for validation are generated from high-level descriptions using a software testing technique. This paper focuses on the interest of re-using and optimizing these validation data for hardware testing purpose. As validation data are already generated when structural test generation begins, we propose to use them as a primary and "free" test set for structural faults. Obviously, to achieve very high fault coverage, this first test set only relying on the validation data will be completed with additional data obtained from a classical gate-level Automatic Test Pattern Generation (ATPG) process. Validation data reuse should decrease the gate-level test generation effort and final test application time.

This paper presents a software testing method for generating validation data. It is based on the well-known mutation testing principle and uses an original sampling technique for decreasing test generation time of these validation data without degrading validation results.

The sequel of this paper is organized as follows. Section 2 presents the classical mutation testing approach. In section 3, mutation operator efficiency is studied. Then, in section 4, we propose a new mutation sampling technique. The paper concludes with section 5.

Originally proposed in [1] as a technique for unit software testing, the aim of mutation testing is to measure the efficiency of a test set to exercise the different functions of a program. This measure can also be used to generate test cases selecting only input data that are mutation adequate. It has been proved in [2] that the data generated by this approach meet most of design validation criteria such as statement coverage, branch coverage, ….

To generate validation data with mutation testing, we select vectors that can distinguish a program from a set of faulty versions of this program, the so-called mutants. These faults, i.e. “small” and syntactically correct modifications of the original instructions, are classified with the help of mutation operators. For VHDL descriptions, a set of ten operators has been defined in [3].

Through mutant simulation, this approach leads to a metric called the Mutation Score (MS). This metric measures the Test Set (TS) quality with respect to a program P. Before to define the MS, lets first define killed and equivalent mutants.

A killed mutant is a mutant that can be distinguished from the original program because it exists at least one data in TS that, when applied on inputs of the original program or the mutant, results in different outputs.

An equivalent mutant cannot be distinguished from the original program whatever the simulated input data.

The mutation score MS is computed as follow:

Where M is the number of generated mutants, K is the number of killed mutants, E is the number of equivalent mutants.

The mutation-based technique is a very time/memory consuming validation technique that must be optimized for complex circuits. A common strategy called mutation sampling consists in selecting a subset of mutants among the whole set of mutants generated from all the mutation operators. At the evidence, if we want to limit the generation effort performed at high level and re-use the validation sequence for a structural test, we must adjust our sampling strategy according to the fault coverage efficiency of the mutation operators. For this, we are going to select all the more mutants generated from one mutation operator than this operator is efficient with the regard to the stuck-at fault coverage.

Because validation data are considered as free data with regard to the detection of stuck-at faults, we compare the so generated test data with pseudo-random test sets generally used as initial test sets before to run an Automatic Test Pattern generation for hard to detect faults. We define metric, called the Non Linear Fault Coverage Efficiency (NLFCE) that allows to take into account the non-linear increasing difficulty to achieve high fault coverage level. This metric considers both the achieved fault coverage and the corresponding test length. First, stuck-at fault simulations performed with validation data on gate-level descriptions deliver corresponding fault coverages: the Mutation Fault Coverage (MFC). In the same way, fault simulations performed with pseudo-random test vectors allow to compute the Random Fault Coverage (RFC). FC% is the relative fault coverage gain between mutation and random data for equal length sequences. L% is the relative length gain between mutation and random data to achieve the same fault coverage. NLFCE is the product FC%. L%.

The experiments are performed on the ITC'99 benchmarks [4] and on the ISCAS'85 benchmarks [5]. These experimental results show that the LOR (Logical Operator Replacement) mutation operator is always the least efficient operator for stuck-at fault detection. Other operators can b

This content is AI-processed based on open access ArXiv data.