Conclusions

This section presents the conclusions of the present research under the following headings: software component testing, software component reliability and robustness, stochastic testing, experimental studies of DBMS, delayed failure, the conceptual framework, and the research problem.

9.2.1 Software Component Testing

a) The conventional approach to software testing is unlikely to detect delayed failures of software components, and alternative approaches should be employed to detect this type of failure. Delayed failures appear to have avoided detection in release testing for both MySQL and Oracle XE, so that faults leading to delayed failure have remained in the production release of software; the testing technique investigated in the present research was able to reveal these failures, and also revealed some known faults in the MySQL DBMS.

b) High-volume stochastic testing using random generation of valid SQL is an effective DBMS testing technique, able to quickly crash or hang the DBMS server, and does not necessarily require the additional effort of generating invalid SQL statements; this was not expected at the outset of the present research.

c) The MySQL DBMS accepts invalid SQL statements, and SQL mutation is an effective test technique for revealing this type of failure, although acceptance of

invalid SQL statements does not appear to have a significantly higher probability of failure (resulting in a crash) than valid statements. The assertion

P2invalid > P2valid that is, the assertion that in the state transition model, the

probability P2 of a transition from S0 to SP is greater for an invalid input than for

a valid input, does not appear to be supported by the experimental results.

d) The mutation operators M1 to M4 as given in BS 7925-2 (2001) are not sufficient to test the cases of swapped element order and repeated elements. The set of mutation operators given in BS 7925-2 (2001) should be extended.

9.2.2 Software Component Reliability and Robustness

a) The present research provides a practical methodology for benchmarking software component dependability (reliability and robustness) and benchmark measures for the MySQL and Oracle XE DBMS have been determined. Metrics for failure delay and reliability for the software components investigated in the study depend on the characteristics of the chosen experimental profile.

b) The MTTF (median F-measure) found in the experimental study was significantly different for MySQL and Oracle XE and this result was unexpected. F-measure values found in the experimental study for both MySQL and Oracle XE were an order of magnitude less than expected.

9.2.3 Stochastic Testing

a) Stochastic testing is an effective method for revealing delayed failure, and is a suitable technique for comparative characterisation of the reliability and robustness of software components; although it is not suited for directly measuring component reliability.

b) The present research provides support for the hypothesis of a saturation effect, as reported by Menzies, Owen & Cukic (2002) and by Menzies, Owen &

Richardson (2007). This effect would make stochastic testing more efficient than might otherwise be expected.

9.2.4 Experimental Studies of DBMS

a) The present research provides experimental results that broadly corroborate related work into testing of DBMS using random generation of SQL, for example as reported by Slutz (1998).

b) Random database creation, in combination with stochastic testing, appears to be a powerful DBMS testing technique. The median of the ratio of F-measure to time constant F/T in cases where failures occurred suggests that failure occurs somewhere between 1T and 2T that is, when between 63% and 87% of database tables have been created.

9.2.5 Delayed Failure

a) Delayed failure is an important failure mode of software components and has been demonstrated at least for the components studied in the present research. There appear to be few systematic studies of delayed failure in the software engineering literature.

b) Experimental demonstration of delayed failure for both ‘valid’ and ‘invalid’ experimental profiles at a 95% confidence level means that the null hypotheses can be rejected, answering the research question in the affirmative by suggesting that the state transition model is valid.

c) Median values for F-measure and failure delay for MySQL were 17.13 and 3.63 blocks respectively. Median values for F-measure and failure delay for Oracle XE were 99 blocks and 60 blocks respectively.

9.2.6 The Conceptual Framework

a) The stateless request-response model provides a useful conceptual framework for black-box testing of software components. Within this conceptual framework, a Markov chain model of error propagation and failure allows calculation of the delayed failure behaviour of software components based on estimation of state transition probabilities.

b) Results of experimental studies of software components (MySQL and Oracle XE DBMS) using stochastic testing are consistent with expected behaviour based on the Markov chain model.

9.2.7 The Research Problem

Experimental demonstration of delayed failure, consistent with long-latency error propagation as described by the state transition model, suggests that the research problem can be solved at least for the DBMS components considered in the study, by using stochastic testing with sequences of test cases at least equal to the measured value of failure delay.