Application Benchmarks

In order to demonstrate further that our PANDA encoding outperforms the native encoding of CadenceSMV for real-life LTL satisfiability checking, we also tested both tools on a

Naive Encodings Model Analysis Times (sec)

DefaultEncodingsModelAnalysisTimes(sec)

10⁰ 10¹ 10² 10³ 10⁴

10⁰ 10¹ 10² 10³ 10⁴

Best encodings with naive vs default variable orders 3-variable, 195 length random formulas

Figure 5.10 : Best encodings of 500 3-variable, 195 length random formulas. Points fall above the diagonal when na¨ıve variable order is best.

MaximumStateSpaceAnalyzed

0 100000 200000 300000 400000 500000

CadenceSMV

PANDA-lexp

3-variable Counter Formulas

Figure 5.11 : Maximum states analyzed before space-out. CadenceSMV quits at 10240 states. PANDA’s NNF/fussy/TGBA/^LEXPscales to 491520 states.

set of application benchmarks, comprised of formulas used to specify actual systems. Our application benchmark formulas come from six sources:⁴

1. acacia demo-v22: 10 formulas 2. acacia demo-v3: 6 formulas 3. acacia example: 25 formulas 4. alaska szymanski: 4 formulas 5. anzu amba: 8 formulas 6. anzu genbuf: 10 formulas

The acacia demo-v22, acacia demo-v3, and acacia example formulas are specifications for systems such as arbiters and traffic-light controllers, distributed with the Acacia tool,⁵ as developed for a study on LTL realizability and synthesis [211]. The alaska szymanski formulas⁶ were developed as liveness properties for the Szymanski mutual exclusion pro-tocol for LTL satisfiability and model checking [53]. The Anzu⁷ benchmarks are sets of formulas used for synthesizing industrial hardware systems from specifications, combined into monolithic formulas for the purpose of satisfiability checking [212]. The anzu amba formulas are specifications for advanced microcontroller bus architectures while the anzu genbuf specifications describe generalized buffers.

We applied PANDA and CadenceSMV to these 63 application benchmark formulas.

PANDA completed 51 formulas before spacing out, while CadenceSMV completed 45

4Thanks to Viktor Schuppan for suggesting these sources, providing some of the formulas in SMV format, and constructing the Anzu formula combinations.

5http://www.antichains.be/acacia/src/acacia_9_linux_i386.tar.gz

6http://www.antichains.be/alaska/tacas08_experiments.zip

7http://www.iaik.tugraz.at/content/research/design_verification/anzu/

model analysis time (sec)

Figure 5.12 : Cactus plot: median model analysis time over all application benchmarks for CadenceSMV and the best PANDA encoding.

formulas before spacing out. The comparison of the performance of CadenceSMV and PANDA’s best encoding on these application benchmark formulas is plotted in Figure 5.12 using a classical cactus plot: the y-axis shows how many instances were checked in time less than or equal to the runtime given on the x-axis, presuming they are run in parallel.

PANDA solved more formulas and did it in less time than CadenceSMV.

5.7 Discussion

This chapter brought attention to the issue of scalable construction of symbolic automata for LTL formulas in the context of LTL satisfiability checking. We defined novel encodings and novel BDD variable orders for accomplishing this task. We explored the impact of these encodings, comprised of combinations of normal forms, automaton forms, transition forms,

and variable orders. We showed that each can have a significant impact on performance.

At the same time, we showed that no single encoding outperforms all others and showed that a multi-encoding approach yields the best result, consistently outperforming the native translation of CadenceSMV.

We do not claim to have exhaustively covered the space of possible encodings of sym-bolic automata. Several papers on the automata-theoretic approach to LTL describe ap-proaches that could be turned into alternative encodings of symbolic automata, cf. [134, 213, 214, 215]. The advantage of the multi-encoding approach we introduced here is its extensibility; adding additional encodings is straightforward. The multi-encoding approach can also be combined with different back ends. In this chapter we used CadenceSMV as a BDD-based back end; using another symbolic back end (cf. [53]) or a SAT-based back end (cf. [216]) would be an alternative approach, as both BDD-based and SAT-based back ends require symbolic automata. Since LTL serves as the basis for industrial languages such as PSL and SVA, the encoding techniques studied here may also serve as the basis for novel encodings of such languages, cf. [217, 204].

While a thorough investigation led us to conclude it is not possible to predict the best encoding for a given formula directly from any statistics that can be gathered from the formula during parse time, this does not exclude the possibility that there may be a dif-ferent way to accomplish this task. It may be possible to investigate how different types of structural properties of specification formulas and techniques for encoding them are re-lated to the efficiency of satisfiability solving or model checking those encodings, similar to the kind of structural analysis that has been accomplished in the propositional satisfiabil-ity communsatisfiabil-ity [218]. It may be possible to use sophisticated machine learning techniques to downselect from the possible encodings a smaller subset predicted to achieve optimal performance [219, 220]. A similar investigation has not yet been done work in the domain of symbolic LTL-to-automata but it is a possible direction for future work. Note that

com-petitive parallel execution strategies similar to the approach presented here are also used to achieve speedups in the propositional satisfiability domain in cases where it is not clear which compilation of a formula will perform best [221].

In this chapter we examined our novel symbolic encodings of LTL in the context of satisfiability checking. An important difference between satisfiability checking and model checking is that in the former we expect to have to handle much larger formulas, since we need to consider the conjunction of properties. Also, in model checking, the size of the symbolic automata can be dwarfed by the size of the model under verification. Thus, the issue of symbolic encoding of automata in the context of model checking deserves a separate investigation.

Chapter 6

Improved Algorithm for Explicit LTL Satisfiability and

Model Checking