General Procedure

The experimental procedure for the 1st experiment with invalid SQL consists of two parts. In the first part of the procedure, as shown in Figure 16, the SUT is initialised and a sequence (or ‘block’) of randomly generated invalid SQL statements is executed, followed by a further blocks of randomly generated valid SQL statements, until the SUT fails. It is assumed that the SUT does fail during the first part of the procedure; otherwise the procedure must be repeated with a different initial seed value until failure occurs.

In the second part of the procedure, as shown in Figure 17, the SUT is re-initialised and only the blocks of valid SQL statements are executed. If a failure of the SUT occurs while executing valid SQL statements in part I of the procedure, but no failure of the SUT occurs while executing valid SQL statements in part II of the procedure, then a delayed failure has been demonstrated. The delayed failure is caused by the SUT accepting a ‘precondition’ invalid SQL statement followed later by a ‘trigger’ valid SQL statement. Block 1 Block 2 … … Block B ↑ Initialise ↑ Invalid ↑ Valid ↓ Fail

Figure 16: Procedure for 1st experiment part I

Block 2 … … Block B ↑ Initialise ↑ Valid ↓ No Fail

The failure delay is then (B−1) blocks or the equivalent number of executed SQL statements. The experiment must be repeated several times to allow for random variation. The median value of failure delay is obtained for several runs of the experiment using different initial seed values for generating random blocks of invalid and valid values, and a 95% confidence interval is calculated.

5.11.2 Procedure for ‘Invalid’ Experimental Profile

The ‘invalid’ experimental profile P1 for MySQL can be generated using either of the

following procedures, corresponding to the two different approaches to SQL mutation.

Procedure (a) for BNF Mutation

Run the following Perl programs:

1. /mysql/programs/bmut.pl BNF mutation

2. /mysql/programs/hgen.pl Generate SQL from BNF 3. /mysql/programs/hparse.pl Parse syntax against BNF

Procedure (b) for Syntax Mutation

Run the following Perl programs:

1. /mysql/programs/hgen.pl Generate SQL from BNF 2. /mysql/programs/smut.pl Syntax mutation

3. /mysql/programs/hparse.pl Parse syntax against BNF

5.11.3 1st Experiment Part I

In the following experimental procedure, a subscripted lower case letter such as s0

letter such as S1 denotes a specific sequence of SQL statements. The first part of the

experiment consists of four steps, as follows:

Step (1) Clear log files. Initialise the SUT to the start-up state s₀. This step ensures that the experimental procedure is repeatable, as the SUT is always started from the same state. It is assumed that the SUT does not fail immediately on start-up otherwise the experiment is inconclusive. For a DBMS such as MySQL the start-up state s0 corresponds to an empty database.

Step (2) Execute a randomly generated sequence of valid SQL statements S1. It is assumed that this causes a sequence of state transitions, leaving the software in an initial state s₁. It is assumed that the software does not fail while executing the sequence of valid SQL statements S₁ otherwise the experiment is inconclusive.

Step (3) Execute a randomly generated sequence of invalid SQL statements S₂ such that at least one of these invalid SQL statements is accepted by the SUT. It is assumed that this causes one or more state transitions leaving the software in a state s2. It is assumed that the software does not fail while executing the sequence of invalid SQL statements S₂ otherwise the experiment is inconclusive.

Step (4) Execute a randomly generated sequence of valid SQL statements S₃. It is assumed that this causes a sequence of state transitions leaving the software in a state s₃. It is assumed that the software fails during this step and so the state s3 is a failed state, otherwise the experiment is inconclusive. A state diagram for the first part of the procedure is shown in Figure 18.

s₀ s1 s2 s3

S1 S2 S3

Figure 18: State diagram for 1st _{experiment part I}

5.11.4 1st Experiment Part II

The second part of the experiment consists of three steps, as follows:

Step (5) Re-initialise the SUT to the start-up state s0.

Step (6) Execute the same sequence of valid SQL statements S₁ as in step (2). This is randomly generated using the same seed value. If the software is deterministic then the SUT will be in the same initial state s₁ as in step (2).

Step (7) Execute the same sequence of valid SQL statements S3 as in step (2). This is randomly generated using the same seed value. It is assumed that this causes a sequence of state transitions leaving the software in a state s4. It is assumed that the software does not fail at this step otherwise the experiment is inconclusive. A state diagram for the second part of the procedure is shown in Figure 19.

s₀ s1

S₁

s₄

S₃