Just as designing a bug-free system is difficult (motivating us to employ model checking to find the bugs), writing correct specifications to check the system is difficult. Therefore, an important step in between writing specifications and performing the model-checking step is specification debugging. As specifications are usually significantly smaller than the systems they describe, they are significantly easier to check. The goal of specification debugging is to answer, as best as possible, the question “do the specifications say what I meant?” Though it is impossible to answer this question absolutely, it is usually suffi- cient to run a series of tests on the set of specifications meant to flag situations where the specifications clearly cannot match their authors’ intentions.
When the model does not satisfy the specification, model-checking tools accompany this negative answer with a counterexample, which points to an inconsistency between the system and the desired behaviors. It is often the case that there is an error in the system model or in the formal specification. Such errors may not be detected when the answer of the model-checking tool is positive: while a positive answer does guarantee that the model satisfies the specification, the answer to the real question, namely, whether the system has the intended behavior, may be different. We need to ask two questions:
1. If there is disagreement between the system model and the specification, which one has the error?
2. If there is agreement, is this caused by an error? (A positive model checking result does not mean there is no error. For example, the specification could have a bug!)
Testing can be performed by specification authors writing small system models that they believe should or should not satisfy each specification and then verifying this is the case via model checking. However, this process can be time-consuming, tedious, and error-prone.
The realization of this unfortunate situation has led to the development of several sanity checks for formal verification [116]. The goal of these checks is to detect errors in the system model or the properties. Sanity checks in industrial tools are typically simple, ad hoc tests, such as checking for enabling conditions that are never enabled [117]. Standard specification testing can also be preformed more intelligently using concept analysis, which produces a hierarchical set of clusters of test traces grouped by similarities [118]. This technique allows the specification author to inspect a small number of clusters instead of a large number of individual traces and use the similarities in the clusters to help determine whether the set of traces in each cluster points to specification error(s) or not.
Of course, it is extremely desirable to run automated tests on specifications. Vacuity detectionprovides a systematic approach for checking whether a subformula of the speci- fication does not affect the satisfaction of the specification in the model. Intuitively, a spec- ification is satisfied vacuously in a model if it is satisfied in some non-interesting way. For example, the LTL specification (req → ^grant) (“every request is eventually followed by a grant”) is satisfied vacuously in a model with no requests. While vacuity checking cannot ensure that whenever a model satisfies a formula the model is correct, it does iden- tify certain positive results as vacuous, increasing the likelihood of capturing modeling and specification errors. Several papers on vacuity checking have been published over the last few years [119, 44, 120, 121, 122, 45, 123, 124], and various industrial model-checking tools support vacuity checking [119, 44, 120].
At a minimum, it is necessary to employ LTL satisfiability checking [52]. If the spec- ification is valid, that is, true in all models, then model checking this specification always results in a positive answer. Basically, the specification is irrelevant. Consider for example
the specification(b1 → ^b2), where b1and b2are propositional formulas. If b1and b2are
logically equivalent, then this specification is valid and is satisfied by all models. Clearly, if a formal property is valid, then this is certainly due to an error. Similarly, if a formal property is unsatisfiable, that is, true in no model, then this is also certainly due to an error. For example, if the specification is(b1∧ ^b2) where b1and b2are contradictory, then the
specification can never be true. Even if each individual property written by the specifier is satisfiable, their conjunction may very well be unsatisfiable. Recall that a logical formula ϕ is valid iff its negation ¬ϕ is not satisfiable. Thus, as a necessary sanity check for debug- ging a specification, we must ensure that both the specification ϕ and its negation ¬ϕ are satisfiable and that the conjunction of all specifications is satisfiable.
Fortunately, this check is easily performed using standard model-checking tools to check the specifications against a universal model before checking them against the system model. A basic observation underlying this conclusion is that LTL satisfiability checking can be reduced to model checking. Consider a formula ϕ over a set Prop of atomic propo- sitions. If a model M is universal,3 that is, it contains all possible traces over Prop, then ϕ
is satisfiable precisely when the model M does not satisfy ¬ϕ. Thus, it is easy to include satisfiability checking using LTL model-checking tools as a specification debugging step in the verification process.4