Related Work

We review literature that has a focus as we have: random simulation methods that are made adaptive and dynamically steered under a specific simulation quality metric.

First, in Figure 4.8, we give a tabular view of the literature and, in particular, which metrics are targeted by the adaptive simulation. Note that in literature, term coverage

metric is used for the same meaning as quality metric in this work. Coverage-directed and

metrics-directed also refer to the same.

Test model preparation

From design specification, e.g. an ISA, construct a Markov chain model 𝑀 = (𝑉, 𝐸, 𝑃) and extend it by attaching weighted constraints to 𝑉;

Initially, all edges and constrains are assigned equal probability/weight; Create meta-mutation 𝑀𝑀 from design under verification;

Start simulation

WHILE still within simulation budget DO

Generate a 𝑡𝑒𝑠𝑡 from 𝑀 and record (𝑡𝑒𝑠𝑡, 𝑒𝑑𝑔𝑒𝑡𝑒𝑠𝑡= (𝑣𝑠𝑡𝑎𝑟𝑡, 𝑣𝑒𝑛𝑑 ), 𝑐𝑜𝑛𝑠𝑡𝑟𝑎𝑖𝑛𝑡𝑡𝑒𝑠𝑡), as

described by Section 4.2.1;

Simulate 𝑀𝑀0 and each activated mutants 𝑀𝑀𝑖 for a cycle with 𝑡𝑒𝑠𝑡 as input, and update

activation statistics of tests, as described by Section 4.2.3; FOR each test has not been updated for a certain time, if any, DO

Calculate and apply an adjustment (𝑃′_{, 𝑊}′_{) on constrained Markov chain model, as}

described by Section 4.2.2. End

END WHILE;

End

Figure 4.7 Summarized procedure for adaptive random test generation directed by mutation analysis.

 The method in [15] begins with a test planning and the coverage is defined as the amount of pre-planned verification tasks that have been simulated, e.g. specific transactions from a CPU unit. It can be viewed as a functional coverage. Then, an evolving Bayesian Network is constructed to model the correlation between test generation directives and the verification-plan coverage.

 In [75], the adaption of random simulation is aimed at exciting more signal-

switching activities at specific locations. Based on the assumption that the increase

of such signal activities in simulation will also lead to higher chances of inciting real design bugs in that portion, the final goal is to improve the efficiency of bug detection, i.e. number of discovered design bugs by a certain number of simulation effort.

Similar to our approach, the random test generation is modeled using a Markov chain. However, no further constraints-based modeling is used as we do. Extra monitors are necessary to be attached to those signals under consideration, so as to collect a weighted score of switch activities. This score is then taken as input to the calculation of adjustment to probabilities on Markov-chain edges.

Besides, the signal switching monitor is extended to also include signals that precede a target signal under observation. They are assigned less weights when summed up into the score, according to their distances to the target signal. This is called depth-driven activity monitoring.

 [16] ( [76] and [77] similarly) builds adaptive constrained simulation based on the so-called observability-based coverage, which we have discussed in the background chapter. Recall that in observability-based coverage, tags are

Metric as adaptation

target Literature

Functional (verification- plan) coverage

[15]: Coverage directed test generation for functional verification

using Bayesian networks (2003)

Functional (signal- switching) coverage

[75]: Microprocessor verification via feedback-adjusted Markov

models (2007)

Observability-based Coverage

[16]: A Functional Validation Technique: Biased Random

Simulation Guided By Observability-Based Coverage (2001)

Assertion based coverage [78] Simulation knowledge extraction and reuse in constrained random processor verification (2013)

introduced as symbolic disturbance to variable values. Their propagation during simulation is defined on logic, arithmetic, and control operators.

It forms a semi-formal method. First, the circuit design itself is modeled as a Markov chain at steady state. The controllability and observability of the nodes, with regard to tags, are estimated using a limited depth re-convergence. Targeting this estimation, an optimization algorithm tries to iteratively perturb the probability distribution on random input generation, each time when the tag coverage stops to increase.

 [78] is one of the most recent effort on coverage-directed constrained random simulation, which targets the assertion-based coverage. An assertions [79] in simulation based verification is simply a statement embedded and co-executed with the “actual” design, asserting whether a specific condition on design state, or a sequence of states is satisfied at that point. Observing shortage of covered assertions, in the verification of a microprocessor, the authors propose a knowledge-learning methodology that tries to extract knowledge during simulation and reuse them to i) further exercise the already covered assertions and ii) generate tests that should hit those un-covered assertions.

A feature based rule learning approach is applied. First, an instruction sequence as test input is converted into multiple snippets, each as a block with equal length. These snippets are classified into two classes (𝑆𝑐𝑜𝑣𝑒𝑟𝑒𝑑 and 𝑆𝑛𝑜𝑡−𝑐𝑜𝑣𝑒𝑟𝑒𝑑, as positive

and negative samples), by whether they covered assertions in simulation or not, according to the simulation trace. Then, ISA dependent features are extracted from the positive samples, such as the data dependences in a sequence of instructions (this is also considered in our approach). Rules are mined, as the knowledge, each representing a hypothetical proposition from a specific collection of features to 𝑆𝑐𝑜𝑣𝑒𝑟𝑒𝑑, i.e. assertions been covered. Concerning the techniques and procedures

used for rule mining, one can directly look into the literature.

Comparison of Literature to Our Work

Contribution from our method compared to other literature on adaptive random simulation can be concluded as follows:

 First, our method uniquely takes the mutation analysis metric as the target of adaptive simulation. Further, based on our consideration that mutation analysis is an advanced emerging quality metric for HDL design simulation, we view our method a step beyond state-of-the-art techniques.

 We employ a combination of constraints and Markov chain to model test generation and enable adaptation, which is not to be found in other methods and

provides us a finer adaptation basis.

 Our adaptation heuristic is based on a unique, more complex test generation problem in mutation analysis: reach, activation, and propagation of mutants. These are aspects that are not covered by other adaptation methods.

To the best of our knowledge, it is the first effort on such mutation-analysis directed adaptive random simulation, to improve the efficiency of HDL mutation analysis. Our evaluation on the method efficiency, in a later chapter, is also based on a state-of-the-art HDL mutation analysis tool: Certitude [21].

4.4. Summary

We have proposed a novel method to improve the efficiency of HDL mutation analysis within constrain random simulation, being aware of the problem that i) initially, the random test model is defined not specifically for a set of mutants and ii) along with the advancing of simulation, un-killed mutants as the remaining target also change. They become the motivation for adaptive random test generation.

The simulation method consists of three parts, for random test modeling and generation, for HDL mutation analysis, and for a consistent adaptation to test generation.

 The Markov chain and constraints based test modeling enables us not only to steer the distribution of a single test input towards our interest, but also to encourage the generation of a certain pattern of two consecutive tests.

 The dynamic mutation schemata leverages the advantage of original mutation schemata by creating and compiling only one meta-mutant. It extends this efficiency by dynamically forking necessary executions of individual mutants and merging them back when the executions succeeded or converged.

 The adaptation heuristic is devised based on the intrinsic problem, or conditions of mutation analysis test generation: reach, activate, and propagate. Basically, test patterns that activated more mutants are encouraged, with the expectation that they will continue to activate many mutant and, therefore, also kill mutants. This encouragement is realized through the adjustment of probabilities/weights on Markov chain edges/constraints.

By this, we expect a derived simulation process that is not only measured under the mutation analysis metric, but also self-steering towards this metric by adaptive, automatic test generation – thus a metrics driven verification method. It severs the first component of our methodology, and the first phase of an IP verification.

In the evaluation chapter, we will mainly investigate whether the adaptation heuristic equipped simulation process is indeed able to improve the efficiency of HDL mutation analysis, i.e. it killing more mutants with less random tests generated.

In general, we see the method not limited mutation analysis, with no restriction of its application to other metrics.

CHAPTER 5: Metaheuristic Search-Based Test