The identification of program variables that should be incorporated by error detection predicates can simplify and guide the design of efficient EDMs. The importance metric, when used in conjunction with an appropriate threshold, can enable this identification process through the generation of an importance ranking of program variables. The application of a threshold to an importance ranking would essentially entail the selection of an importance value or ranking position, where all program variables with an importance metric value greater than the selected value or occupying a position greater than the selected position would be considered critical variables. Such an application of the importance metric would allow software engineers to precisely target those variables which have the most significant impact upon the correct functioning of a software system. For example, from Table 4.12 it can be observed that program variables processedP osition and remainLen have the highest importance values. If a threshold were set such that these program variables could be considered critical then it would follow that an EDM must ascertain that these program variables hold appropriate values during the execution of the associated software system. The relative rankings generated by the importance metric can also be used
4. Towards the Identification of Critical Variables
to focus the efforts of software engineers on specific software modules. For example, from Tables 4.10-4.12 it can be seen that program variables associated with software modules 7Z1, FG2 and MG2 feature heavily in the importance rankings for their respective software systems, suggesting that these software modules should be equipped with dependability mechanisms, re-engineered or closely monitored. This use of the importance metric is particularly applicable in the context of commercial software system development, where subsequent software system releases can not address all known issues and, hence, software engineers will seek to address software vulnerabilities in order of severity. In such a situation it would be possible to threshold the relative rankings generated as part of a cost-benefit analysis, thus ensuring that the most severe vulnerabilities are readily addressed. Further, observe that the relative rankings presented in Tables 4.1-4.12 have been generated without prior knowledge of software system structure or functionality. As the importance measure does not take into account composition or communication paths, it can be readily applied in situations where dependability is to be assessed post-implementation or by software engineers who were not directly involved in the implementation of a software system. Again, this is consistent with current approaches to commercial software engineering, where dedicated teams often deal with the maintenance and on-going support for previously developed software systems.
Over 38 million fault injection experiments were performed in order to es- timate values for failure rate, spatial impact and temporal impact of program variables in all target software modules. The PROPANE fault injection suite allowed these experiments to be conducted in an automated fashion. Provided that the instrumentation of the target system is concerned with maximising the number of program variables instrumented in a given module, which will gen- erally be the case for most software systems, there is no reason why this level of automation can not be achieved for any given target software system. Once the values for the spatial and temporal impact metrics have been determined, the importance metric can be automatically calculated for each instrumented
4. Towards the Identification of Critical Variables
variable. As the definition of the importance metric is fixed for a software sys- tem, rather than individual variables or modules, the cost of employing the proposed approach, in terms of engineering effort, is relatively low. Further, in the case of dependable software systems, the fact that fault injection analysis is a commonly adopted dependability validation techniques means that the in- formation required to evaluate the importance metric is likely to be available during software system development.
There are limitations associated with the use of the importance metric. In particular, the relative ranking generated by the importance metric is sensitive to the set of program variables under consideration. Ideally, all variables within a software module should be analysed, thus ensuring that no variable with a potentially high importance, which may subsequently lead it it being considered a critical variable, is overlooked. However, performing such a comprehensive analysis may be impractical in some situations, as it may require a prohibitive number of fault injection experiments, particularly if all the software modules in a software system are to be analysed. However, this limitation is, to a large extent, mitigated by the extended validation and testing periods associated with dependable software systems and the automated nature of the analysis process. Another perceived limitation of the importance metric is that its value for a particular program variable is not an objective representation of the real- world importance of that program variable. Rather it is a value which, when viewed relative to others produced during the same analysis, can focus the efforts of dependability engineers and allow a meaningful cost-benefit analysis to be undertaken. Finally, although it is not a directly limitation of the metric itself, the fault injection analysis approach proposed for estimating the importance metric suffers from the inherent limitations of the fault injection process, such as a dependence upon a the identification of a representative set of test cases. As these are limitations of the evaluation mechanism, and more generally of fault injection analysis as a dependability validation technique, they do not negatively impact on the use of the importance metric as applied in this thesis.
4. Towards the Identification of Critical Variables