This section discusses how to use CLP for program generation. We will begin with a syntactic description of a simple arithmetic expression language and express progres- sively more interesting properties for expression generation. Section 2.3 will then discuss applying these ideas to program generation specifically for JavaScript.
2.2.1
Syntactic Expressions
We use the arithmetic expression language shown in Figure 2.1 for the examples throughout this section. We can describe this grammar in CLP, as shown in Figure 2.2.
i ∈ Z
e ∈ ArithExp ::= i|e1+ e2
Figure 2.1: Basic arithmetic expression language, consisting of integers and addition over expressions.
As shown in Figure 2.2, two clauses are used in the exp procedure, one for each of the alternative productions of ArithExp in Figure 2.1. The first clause (on lines 1-3) checks that the input (I) is an integer between INT_MIN and INT_MAX, where INT_MIN and INT_MAX are assumed to be defined elsewhere with the obvious meanings. The second clause (on lines 4-6) checks for the case of e1+ e2, which is assumed to be represented as
1 exp ( I ) :−
2 INT_MIN #=< I , 3 I #=< INT_MAX. 4 exp ( add (E1 , E2 ) ) :− 5 exp (E1 ) ,
6 exp (E2 ) .
Figure 2.2: CLP implementation of the grammar shown in Figure 2.1
e1 (E1) and e2 (E2) should themselves be expressions, exp is recursively used on these
parameters to ensure that they are, in fact, expressions (on lines 5-6).
We can use the CLP definition in Figure 2.2 in two distinct ways: to recognize valid expressions and to generate valid expressions. In the recognition case, we pass a potential expression as an argument to the exp procedure and it will return either true if it is a valid expression or false otherwise. The more interesting case for fuzzing is generation: if we want to generate valid expressions instead, we can use a query like the following:
1 ?− exp (E) , 2 write(E) , 3 f a i l .
With the above query, the CLP engine will first attempt to find a value for the logical variable E that will make the exp procedure true on line 1. Once such a value is found, the engine will write that value to output on line 2, then fail on line 3. Failure will cause the engine to backtrack and attempt to find a different value for E; this process will continue indefinitely and generate an infinite stream of valid expressions.
One caveat is that the resulting expressions will not be concrete; instead, they will contain symbolic variables (standing for unknown integers) that are subject to a set of constraintsderived from the clauses. For example, one of the generated expressions would be add(X, add(Y, Z)) where X, Y, and Z are symbolic variables standing for unknown integers. The CLP engine remembers the bounding constraints on each one of these
variables (i.e., ∀a∈{X,Y,Z} INT_MIN ≤ a ∧ a ≤ INT_MAX). To get a concrete expression, we
need to then ask the CLP engine to label the symbolic variables, that is, find concrete values that satisfy the constraints. The engine guarantees that satisfying values must exist.
2.2.2
Bounding Size
CLP engines use unbounded depth-first search by default. This strategy ultimately controls the order in which expressions are generated. For an infinite stream of expressions (as would be produced by the generator in Figure 2.2), the search strategy controls exactly which expressions are generated within a finite amount of time. While the search strategy is effectively “baked-in” the engine, we can still exercise simple control over it. For example, consider the following query:
1 ?− call_with_depth_limit ( exp (E) , 5 , CurrDepth ) , 2 CurrDepth \== depth_limit_exceeded ,
3 write(E) , 4 f a i l .
The above query uses call_with_depth_limit on line 1, which is a built-in procedure in the SWI-PL [67] CLP engine. This procedure runs a given query (exp(E) above) with a bound on the recursion depth (5 above). If the query ever exceeds this bound, it will unify a given variable (CurrDepth) with the atom depth_limit_exceeded. This can then be used as a flag to check whether or not the query exceeded the bound, as is checked in line 2 above. The code on line 2 will cause failure to occur if the bound was exceeded (i.e., this checks to see that CurrDepth does not hold the atom depth_limit_exceeded). If the depth is not exceeded, CurrDepth will instead hold an integer indicating how deep the recursion went. With all this in mind, the above query effectively bounds the depth of the expressions generated.
desired. Using iterative deepening, we can generate minimal test cases which find bugs. This is in stark contrast to the bulk of related work, which relies on random generation of large programs (e.g., [14, 13, 12, 21, 74]). While such large programs find bugs, they tend to contain lots of unrelated code, necessitating downstream test case reduction techniques (e.g., [75, 76, 77, 78, 79, 80, 81, 82]).
While the bounding approach described here based on call_with_depth_limit works for stochastic grammars, it is tied to SWI-PL [67]. Additionally, it does not scale well to larger problems, and it is difficult to bound based on elements other than recursion depth without significant code changes. Chapter 9, Section 9.2.2 describes this problem in detail. Chapter 9 also offers a general solution in Section 9.4.1.
2.2.3
Stochastic Grammars
CLP subsumes the stochastic grammar technique for program generation. This means that we can specify stochastic grammars using CLP, and do so quite easily. An example showing such stochastic behavior is shown in Figure 2.3, which was derived from the CLP code in Figure 2.2. The key to the stochastic behavior in Figure 2.3 is the use of the maybe procedure, which triggers random failure with a given probability. This leads to a somewhat random distribution of programs being produced, as is expected from stochastic grammars.
It should be noted that while the maybe-based technique described is generally suffi- cient for stochastic grammars, this tends not to scale well with more complex generators. The reasons why, along with solutions to this problem, are discussed in Chapter 9, specif- ically Sections 9.2.1 and 9.4.3.
1 exp ( I ) :− 2 maybe ( 0 . 4 ) , 3 INT_MIN #=< I , 4 I #=< INT_MAX. 5 exp ( add (E1 , E2 ) ) :− 6 exp (E1 ) ,
7 exp (E2 ) .
Figure 2.3: CLP code from Figure 2.2 augmented with stochastic capabilities, thanks to the built-in maybe procedure in the SWI-PL [67] CLP engine. The use of maybe above triggers random failure with 40% probability, allowing for random exploration similar to that of a stochastic grammar.
2.2.4
Arithmetic Overflow
One of the most powerful abilities of CLP is symbolic arithmetic reasoning, which enables the user to specify numeric constraints as part of a predicate. These constraints are handled by an integrated constraint solver as part of the CLP engine. For example, we can specify that we only want to generate expressions that contain at least one arithmetic overflow, shown in Figure 2.4. Stepping through this code, the exp procedure (derived from Figure 2.2) has been instrumented with a second integer parameter holding the result of the given expression, along with a third boolean parameter indicating whether or not overflow has occurred. Lines 5-7 are almost identical to lines 1-3 in Figure 2.2, except that now exp indicates that overflow did not occur in line 5. Lines 8-16 are more interesting, because these dicate how arithmetic addition is performed. Line 11 performs the addition, though in a symbolic way with #=. If the result of this operation is greater than INT_MAX, then overflow (Over) is set to true in line 12. Similarly, if the result of this operation is less than INT_MIN, then overflow is set in line 13. If, however, the result is between INT_MIN (line 14) and INT_MAX (line 15), we do not immediately set that overflow occurred. Instead, this will return true (indicating overflow occurred) if either of the subexpressions overflowed, using the boolOr helper prodedure. Only if the result
was between INT_MIN and INT_MAX, and if none of the subexpressions cause overflow to occur, will the result of the addition indicate overflow.
1 boolOr ( true , _, true ) . 2 boolOr (_, true , true ) .
3 boolOr ( false , false , false ) . 4
5 exp ( I , I , fa lse ) :− 6 INT_MIN #=< I , 7 I #=< INT_MAX.
8 exp ( add (E1 , E2 ) , N, Over ) :− 9 exp (E1 , N1 , Over1 ) ,
10 exp (E2 , N2 , Over2 ) , 11 N #= N1 + N2 ,
12 ( (N #> INT_MAX, Over = true ) ; 13 (N #< INT_MIN, Over = true ) ; 14 (N #>= INT_MIN,
15 N #=< INT_MAX,
16 boolOr ( Over1 , Over2 , Over ) ) ) .
Figure 2.4: CLP code snippet which can generate programs exhibiting arithmetic over- flow. The second parameter of the exp procedure indicates the integer result of evaluating the arithmetic expression. The third parameter of the exp procedure dictates if overflow occurs in the expression generated (the first parameter to exp), with true indicating overflow occurred and false indicating otherwise. This is based on the CLP code in Figure 2.2.
An example query using the code in Figure 2.4 to generate expressions that contain at least one overflow is shown below:
1 ?− exp (E, _, true ) , 2 write(E) ,
3 f a i l .
The above query will generate some expression E which is required to overflow somewhere, as indicated by the third parameter to exp being true in line 1. In this case, we do not care about the actual result of the expression, indicated by the use of underscore (_) for the second parameter to exp in line 1. It will then write out the expression on line 2 and
trigger backtracking for another expression on line 3, much like the query shown for the original CLP code in Figure 2.2.