6.2.1 Instantiation
The HAZOP method requires the identification of so-called guide words that are used in a questioning process to identify deviations from the design intent. For example, "less" and "late" would prompt consideration of what would happen if the system output was less than expected in some sense or later than expected.
For integrity and availability attacks, these guide words can be instantiated with questions relating to deviations from the expected integrity and availability. The HAZOP technique does not add value for, and is inappropriate for the consideration of confidentiality attacks.
The adversarial goals offered by Papernot[76, p. 9] provide a categorisation for
integrity attacks, into confidence reduction, misclassification, targeted misclassifica- tion and source/target misclassification. Unfortunately, no argumentation is offered
ments offered that the categories are exclusive. Despite these concerns, this categori- sation could be used to model the possible outcomes where differences in the type of deviation are important in Industry 4.0. For example, if consideration should be given to the effect on the industrial processes "if there is a confidence reduction" in the output of the machine learning system.
An improvement is also to include in the analysis and in the HAZOP table pro- duced, a consideration of what the outcome would be if the deviation in integrity was detected and also if not detected, in terms of the likely consequences. As previ- ously mentioned, some integrity attacks are designed to be overt and will be difficult to detect. Others may well cause deviations significant enough to be detected, either via decision verification during evaluation or by supervision of the business process. Regarding availability, the HAZOP method is already fit for the consideration of deviations from availability. Appropriate guide words can be used to focus on both the machine learning system rate of output and the time of unavailability (e.g. rate of classification less than 5 classifications per second and classifications not available during weekends and other non-essential work periods).
These instantiations of the questions to machine learning is crucial, because of the need in the risk management to understand in which ways and to which degree the deviations might happen as a result of an attack. The value added here is similar to a domain specific checklist being incorporated in the CORAS method. Moreover, the instantiation here adds value in that the machine learning instantiation of questions is that it both focuses the thinking on the integrity and availability of the output and the different corrupted outputs.
There are other limitations, identified in the literature, of HAZOP that are rele- vant in this domain of machine learning and Industry 4.0:
1. An inability to handle interactions between different parts of a system or pro- cess. However, this inability is covered by the Dependency Modelling in Sub-
section5.3.5.
2. An inability to assess the effectiveness of defences or mitigations. This assess- ment is difficult in any event, given the current level of understanding regard- ing machine learning systems. However, the assessment of the effectiveness of mitigations is modelled in CORAS and the Security Assessment Method. 3. An inability to assess interactions between different parts of a system or pro-
cess. This remains a concern for complex applications and reinforcement learn- ing applications in Industry 4.0.
6.2.2 HAZOP and Rapidly Changing and Evolving Systems
As HAZOP tables are relatively simple, the tables can be updated in line with changes in systems incorporating machine learning systems, in mitigating controls where the changes in the risks and consequences are material enough to warrant this. Mitigat- ing controls can be cross-referenced with treatments tracked in CORAS above to maintain consistency and a coherent track record of changes.
We then turn to the development of the Dependency Modelling as a complemen- tary method to CORAS in the risk management framework.
No Guide Word Element Deviation Possible causes Consequences Safeguards Actions Required Assign each entry a unique tracking
number Insert deviation guide word
Describe what the guide word pertains
to Describe the variation
Describe how the deviation may occur Describe what may happen if the deviation occurs List controls (preventative or reactive that reduce deviation likelihood or severity Identify any hazard mitigation or control actions required 1 Confidence Reduction
Classifier Reduce the output confidence classification Poisoning of test data Poisoning of test data (label attack) Impersonate attack
No Effect General Controls (See Standard) Special Controls (See Standard)
Accepted
2 Misclassification Classifier Alter the output classification to any class different from the original class Poisoning of test data Poisoning of test data (label attack) Impersonate attack
As above As above Accepted
3 Targeted
Misclassification Classifier Produce inputs that force output classification into a specific target class. Poisoning of test data Poisoning of test data (label attack) Impersonate attack
As above As above Accepted
4 Source/Target Misclassification
Classifier Force the output classification of a specific input to be a specific target class. Poisoning of test data Poisoning of test data (label attack) Impersonate attack
As above As above Accepted
5 No operational (overloaded) Available, but responses out of acceptable time bands Denial of Service Attack
As above As above Accepted
6 Not operational Classifier No Service provided
Reprogramming Attack
As above As above Accepted
FIGURE 6.4: Extract of Key Fields from the Table, including Guide Words for Machine Learning Systems