Threats to Validity

● symbolicJ stack caching baseline

0 5 10 15 20 25 30

0e+001e+052e+053e+05

Time (min)

Number of Explored States

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

● symbolicJ stack caching baseline

0 5 10 15 20 25 30

0500000100000015000002000000

Time (min)

Number of Explored States

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

● symbolicJ stack caching baseline

Fig. 7.9: Average number of states explored from 5K subjects using CVC4, MathSAT and Z3 in 30 min.

7.3.4 Threats to Validity

As usual, it is possible that results do not generalize much beyond our subject set. To mitigate this threat we used a set of 300 automatically-generated Java programs and a set of 96 real C programs from the GNU operating system.

Besides, it is possible that results are specific to Z3. To address this threat we considered other solvers, namely CVC4 and MathSAT5. Although these solvers show different costs for different constraint problems, we observed a similar behavior on these solvers for the techniques we analyzed.

Another threat to validity is the possibility of errors in our implemen-tation. We carefully inspected our code and the consistency of our results.

Nevertheless, additional experiments are necessary to assess generality of our results.

7.4 Related Work

Symbolic execution is a technique for systematic test-input generation that has gained significant momentum in recent years [Cadar et al., 2011]. Several tools have been proposed to support symbolic execution, including Bogor/Ki-asan [Deng et al., 2006], Cloud9 [Bucur et al., 2011], (j)CUTE [Sen et al., 2005], DART [Godefroid et al., 2005], DiSE [Yang et al., 2014], DSC [Islam and Csall-ner, 2010], EXE [Cadar et al., 2008b], eXpress [Taneja et al., 2011], KLEE [Cadar et al., 2008a], ParSym [Siddiqui and Khurshid, 2010], PEX [Tillmann and de Halleux, 2008], SAGE [Godefroid et al., 2012], SMART [Godefroid, 2011], and SPF [Pasareanu and Rungta, 2010]. Unfortunately, symbolic execution is expensive both in time and space. We discuss most-related recent work to reduce the high cost in constraint solving and in path exploration during symbolic execution.

Time Reduction

Typically, symbolic execution spends a major part of its time in constraint solving. Cadar et al. [Cadar et al., 2008a] proposed several approaches to simplify constraints prior to calling an SMT solver during symbolic execution.

The static symbolic execution tool KLEE implements Caching as we described.

In addition, KLEE implements constraint checking with a potential solution.

It is based on the assumption that a solution of subset often satisfies extra constraints. We need to investigate how this additional optimization compares with those we considered.

Visser et al. [Visser et al., 2012] proposed GREEN, an infrastructure to share results of symbolic executions across different environments. It not only reuses the previously results in a single symbolic analysis, but also reuse more generalized results such as the results from other runs of symbolic executions and the results from different users. Moreover, they reused the results for blocks of complicated code. GREEN proposes canonical representations of path conditions to enable caching across different programs. The intuition is that after partitioning constraints with respect to dependent clauses the chance of finding structurally equal symbolic constraints increases. For example, solutions to constraints produced in the symbolic execution of one program could be used to solve constraints produced from the symbolic execution for another program. The results of GREEN are encouraging to speedup constraint solving. It is important to note that we also optimized the caching algorithm by incrementally constructing the independent constraints, instead of constructing them from scratch each time.

7.4 Related Work 129 Incremental SMT solving is an active field of research with the goal of optimizing problems that can be characterized by many similar sub-problems.

For example, detecting the longest program execution trace [Li et al., 2014], solving scheduling problems [Steiner, 2010], etc. As a basic decision procedure, incremental SMT solving searches for a satisfying assignment by performing various operations (e.g., unit propagation). When internal conflicts occur, incremental SMT solvers extract and store conflict clause to prune exploration search space. More specifically, incremental solvers store learned and conflict-ing clauses in the assertion stack so that they can be reused upon backtrackconflict-ing.

Recently, Audemard et al. [Audemard et al., 2013] proposed a technique to strengthen the clauses learned by the solver by extending an incremental SMT solver to execute in multiple threads. We observed that this development can directly improve symbolic execution.

Incremental SMT solving [Hooker, 1993; Whittemore et al., 2001; Wieringa, 2014] has been applied in some domain and achieved higher performance by incrementally solving sets of related problems. Some experimental evi-dence showed the effectiveness of the incremental facility to classical solvers.

For symbolic execution, to the best of our knowledge, there was no existing symbolic execution tool using incremental SMT solving before our related publication [Liu et al., 2014]. We evaluated the benefits of the gain and recom-mend the use of incremental SMT solving in symbolic execution.

Space Reduction

Several techniques have been proposed to address the path-explosion problem in symbolic execution. Godefroid [Godefroid, 2007] proposed compositional symbolic execution. The idea is to perform symbolic execution using sym-bolic summaries of functions that are incrementally computed in symsym-bolic execution. Yang et al. [Yang et al., 2014] proposed Directed Incremental Sym-bolic Execution (DiSE) to speedup symSym-bolic execution by capitalizing on the changes across two code versions. Taneja et al. [Taneja et al., 2011] proposed eXpress that builds on similar ideas. Both compositional and differential symbolic execution are legitimate approaches to scale symbolic execution.

SymbolicJ is orthogonal to these techniques. It improves symbolic execution on the apace dimension by eliminating common sub-expressions and building the path conditions prior to path exploration. It is also possible to customize SymbolicJ to perform compositional and differential symbolic execution.

Anand et al. [Anand et al., 2007a] proposed the use of an interprocedural static analysis to detect potential flows of symbolic data to statements. One of the goals was to speedup symbolic execution by avoiding unnecessary code instrumentation. Even though SymbolicJ does not require code instrumenta-tion we plan to evaluate how SymbolicJ compares to with further optimizainstrumenta-tion applied. Also, Siddiqui and Khurshid [Siddiqui and Khurshid, 2010], Staats and Pˇasˇareanu [Staats and Pasareanu, 2010], and Bucur et al. [Bucur et al., 2011] independently proposed the use of Parallel Symbolic Execution (PSE) to

optimize the time dimension for symbolic execution. SymbolicJ complements parallel symbolic execution. For example, our approach can be used on each of the parallel processes to amplify the improvement of PSE.

More recently, Anand and Harrold [Anand and Harrold, 2011] proposed Heap Cloning. The idea of the technique is to use an image of the concrete heap while performing symbolic execution. One of the goals was to speedup symbolic execution by avoiding unnecessary calls to reliable library code.

Park et al. [Park et al., 2012] proposed the use of large test suite bases to support constraint solving. They use DSC [Islam and Csallner, 2010], a test case generator using dynamic symbolic execution, to build path conditions and existing test inputs to solve constraints. We will evaluate how SymbolicJ compares to optimized versions of DSC that use such techniques.

7.5 Conclusion

Program verification techniques using symbolic execution generally check the model of a program for the desired property by proving that all paths to certain error nodes are infeasible (i.e., no program execution will violate the property). One important obstacle for gaining high structural coverage is the high cost of path condition solving.

We present an incremental SMT solving (stack-based) approach for symbolic execution. Our approach represents the program under analysis in a decision graph with respect to the loop bounds and constructs the path conditions (concerning the class bounds) before the path exploration. Thus it reduces the time of path condition generation. In the phase of path exploration, our approach reduces the time of path condition solving by eliminating common sub-expressions in the conditions and exploiting the recent advances of in-cremental SMT solvers. An inin-cremental SMT solver reuses the intermediate lemmas that it has learned in previously constraint solving in contrast to a common SMT solver that learns lemmas from scratch each time it is invoked.

We have implemented our approach in the prototype tool SymbolicJ. To the best of our knowledge no existing symbolic execution tool uses incre-mental SMT solving for symbolic execution. Hence, it is important to eval-uate how helpful our approach can be. We performed an empirical study on the efficiency of symbolic execution using different techniques. We com-pared SymbolicJ with various alternatives using incremental solving. We considered various options of cache-based incremental solving and a large set of programs; both real (96 C programs from the KLEE benchmark) and artificially-generated (300 randomly-generated programs of various sizes: 5, 10, and 20K). Overall, results indicate that stack-based approaches provide superior results compared to cache-based approaches. The median speedup obtained when using the support of a modern incremental SMT solver is of

∼ 5x (min.:∼ 1x, avg.:∼ 4.8x, max.: ∼ 9x). When the optimization that elim-inates common sub-expressions is enabled in stack-based approach (i.e., the

7.5 Conclusion 131 approach provided by SymbolicJ), results indicate that the speedup obtained is of ∼2.57x compared to a basic stack-based approach. Note that results are restricted to the use depth-first search. More research is needed to find ways to combine caching- and stack-based approaches to improve results even further.

Part V

Conclusion

CHAPTER 8