Efficacy is the degree to which the artefact produces its desired effect.[108, p. 5] The
primary evaluation of the efficacy of the risk management framework is the case study presented in this section. Limitations, both known and drawn from the case study are also considered in this section. Due to the sensitivity of the risk assess- ment, any information identifying the company ("the Company") involved has been obfuscated.
Description of the Use Case in the Case Study The machine learning system ("the ML system") chosen is in a cloud platform incorporating many data analytics sub- systems ("the Platform"). The use case is a factory (’the Factory") performing manu- facturing tasks including the assembly of system boards. One of the planned appli- cations or use cases of the ML system is a cyber-physical quality use case whereby system boards are checked optically to detect scratches, fingerprints, chassis defects and missing screws binding sub-components on the system boards in one factory manufacturing line. The optical check is performed at the end of the assembly, prior to packaging. The detection of an abnormality results in a board being added to the abnormal flow by a Manufacturing Execution System ("the MES").
There is a low rate of abnormalities, estimated at 3 per cent of boards. Before the Use Case being put into effect, manual checks of quality are performed, but only with sampling and not every finished board, unlike the Use Case.
The CORAS method provides for eight steps to be followed in the risk assess- ment. To allow for the HAZOP and Dependency modelling, these steps were ex- tended to ensure these were included in the case study. As well, the Derived Threat
Actors (Fig. 2.3 Derived Attacks (Fig. 2.1) and Derived Defences (Summary taken
from Subsection2.3.9) were incorporated as input, as shown in Table8.1. Derived
Mitigations were included earlier, so as not to prolong the analysis unnecessarily.
Company Objectives For the Company, the objectives are not primarily to have a risk assessment of the Use Case, but rather, to gain knowledge of machine learning threats, how they may manifest in the applications of the ML system and in the Factory, and the gain knowledge and tools to perform risk management.
Consistent with the objectives of this research, the objective was to undertake an end-to-end analysis, incorporating a business process perspective, and at a high- level of abstraction.
Factory. The Stakeholders participating in the Case Study and Risk Assessment in- clude:
1. Factory Business Process Representatives 2. Factory IT Architects
3. Platform Business Owner 4. Platform Architects 5. Data Analysts
6. Integration Program Manager
8.1.1 Case Study Results
The deliverables of the Risk Assessment were:
1. Approved Risk Matrix (See TableB.3)
2. Approved Risk Acceptance Matrix (See TablesB.1andB.2)
3. Approved CORAS Diagrams (See FiguresB.2,B.3,B.4,B.5andB.6)
4. Approved HAZOP Table (See TableB.4)
5. Approved Dependency Diagram (See DiagramB.7,B.8andB.9)
These outcomes can be summarised as confirmation that the risks were accept- able without additional controls or treatments being required. In interpreting the case study results, this is not surprising as:
1. The limited consequences arising from only a loss of quality. The ability of the Factory to revert to a less effective manual quality check lessens the reduction in quality from a compromised ML system. The highest impacts are a loss of security reputation for the ML system and because the Company is in the business of providing security services to clients as well as the Platform. 2. The limited consequences of images of system boards being made available to,
for example, competitors. This information is available if a laptop, for example, is simply opened.
3. Reductions in integrity would be detected: The capability to revert to human inspections if the accuracy of the ML system would decrease. An increase in false positives (boards seemingly defective, but OK) would be noticed when investigating the abnormalities further. An increase in false negatives (i.e. missing abnormal boards) decrease would likely be noticed in a final, out of the box general quality inspection or worst case feedback from purchasers of the system boards after several days.
4. There are existing controls in place to protect the training data, the evaluation data, the output from the ML system and attackers accessing the ML system itself.
CORAS Steps
CORAS Actions
Additions to CORAS
Method Additional Input to Step Output from Step
Step 1
Introductory Meeting
Series of preliminary meetings to establish that Company Goals and Research Goals are aligned and chose a suitable use case
Presentation Confirming Objectives and Scope of the Case Study List of Stakeholders/Participa nts Step 2 High Level Analysis Introduction to Stakeholders - Derived Attacks Introduction to Stakeholders - Threat Actors Introduction to Stakholders - Dependency Analysis
High Level Asset Diagram
High Level Activities Diagram
Step 3 Approval
Risk Categorisations (Likelihood and Probability)
Risk Acceptance Matrix
Step 4 Risk Identification
Derived Threat Actors
Derived Attacks
Derived Mitigations
Pre-prepared CORAS Risk Diagrams based on these inputs, outpus Step 2 Step 5 Risk Estimation Step 6 Risk Evaluation Pre-prepared Hazop Table Pre-prepared Dependency Diagram HAZOP Table Dependency Diagram Step 7 Risk Treatment Derived Mitigations Pre-prepared CORAS Treatment Diagrams Hazop Table (treatments added) Dependency Diagram (extended to include treatments) Step 8 Conclusions CORAS Treatment Diagrams HAZOP Table Dependency Diagram
Approved Risk Matrix
Approved Risk Acceptance Matrix
Approved CORAS Diagrams
Approved HAZOP Table
Approved Dependency Diagram
learning attack.
6. The lack of a motive for an attacker to carry out an attack if the consequences are low. An attack on the Factory or the Platform would likely be in the form of an easier attack to mount with more of reward for the attacker (e.g. stealing commercial data, causing more Factory disruption)
In conclusion, the risk management framework promoted the objectives of risk management, primarily in relation to the improved identification of opportunities and threats and providing a confident and rigorous basis for decision making and
planning. (See Subsection3.1.1.) However, many of the objectives have not been
demonstrated in the case study, such as actual loss reduction, enhanced safety, due to the limitations of the case study and the need to demonstrate and measure such achievements over time.
8.1.2 Expert Review
Expert reviews undertaken include the evaluation comment that:
"In general I think that your selection of methods and techniques can work effectively to assess the cyber risk. CORAS for overview and struc- ture and the other techniques as a refinement to make it ML/ind4.0- specific. Predefined tables, such as the one based on HAZOP, make it not only more ML/ind4.0 specific but also support an effective thinking process."
[24]
8.1.3 Further Argumentation
The use of catalogues has been demonstrated to increase the efficacy of a risk as-
sessment methodology.[59, p. 46] On the other-hand, the quality (completeness and
accuracy) of the catalogues have a direct influence on whether the risk management
framework produces accurate results. (See Subsection8.4)
Risk Management Framework Evaluation - Limitations and Future Research
CORAS Scalability Perhaps the most critical limitation of the CORAS method is its scalability and this was observed in practice in the case study. Although pow- erful, the visual representation of risks can quickly become unmanageable is one diagram when using CORAS in any complicated domain. In the context of machine
learning, the number of attacks with differing pre-conditions (See Fig.2.1) the num-
ber of likely assets involved in the data flow and the range of treatments to consider, including tradition IT controls such the secure transfer of information mean that a simple use case will generate a very large diagram. Some practical steps can be taken, for instance, to divide the diagrams generated up into diagrams per attack. But this comes at the cost of clarity and manageability of the risks.
An alternative is to model the risks in a tabular manner. See, for example, [59,
p. 72]. Tables of modelled risks are more useful at scale as they allow for better manipulation. A hybrid approach can combine the best of visual approaches, such as CORAS, with a tabular approach. CORAS already makes use of tables, such as in
the high-level design, and converting diagrams to tables is possible. Further work here is needed to determine how this could be applied to the CORAS method.
As per CORAS, the dependency modelling suffers from similar scalability issues. It too could benefit from a hybrid, visual/tabular approach. However, due to its top-down approach, unlike CORAS, there may be less need to have very in-depth diagrams. However, further research is necessary.
Other Limitations As identified in Subsection5.2.2, CORAS is not able to model the dependencies between resources, such as business processes impacted by a risk. However, it is difficult to imagine a scenario where this might be a serious concern and also may be covered by the dependency modelling. Other risk management objectives, such as the effective allocation and use of resources for risk treatment, remain beyond the capabilities of a risk management framework. (See Subsection
3.1.1.)
Also, as noted, although the risk management framework allows for joint risk assessment activities by different organisations involved in the application of ma- chine learning in Industry 4.0, there are disincentives that may well prevent such
joint activities. (See Subsection3.2.9.)
Limitation - Full Analytical Model of Adversary’s Behaviour Although con- ceptually simple, the security evaluation assessment process requires a full analyti- cal model of the problem and of the adversarys behaviour, that may be very difficult
to develop for real-world applications in Industry 4.0.[19, p. 13] The threat actor
analysis in this paper may go some way to providing that model, but an empirical assessment of whether it will work in practice, such as in a case study is still neces- sary.
Limitation -Penetration Testing and Red Team Exercises The penetration test-
ing in Subsection 5.3.2 may be carried out in the typical fashion by penetration
teams, examining the state-of-the-art in machine learning attacks and choosing suit- able attacks for the machine learning system in question. Unfortunately, such com- mercial services for penetration testing of machine learning systems are not available at the time of writing.