The Design of a Risk Management Framework for Machine Learning Systems in Industry 4.0

(1)

The Design of a Risk Management

Framework for Machine Learning Systems

in Industry 4.0

Author:

Antony HIBBERT

Supervisor: Associate Professor Jan C.A. van der Lubbe and Dennis de Geus RE RA CISA

A thesis submitted in fulfilment of the requirements for the Master’s Degree

(2)

(3)

Abstract

Antony HIBBERT

The Design of a Risk Management Framework for Machine Learning Systems in Industry 4.0

Machine learning is a prevalent Artificial Intelligence technique that is being imple-mented in industry, as part of Industry 4.0 - the so called fourth industrial revolution introducing internet technologies to industry. In industry, machine learning appli-cations include industrial design, process control, robotics and assembly. Though machine learning offers clear benefits to industry in terms of automation and ef-ficiency improvements, these machine learning systems are susceptible to a broad range of attacks that can lead to the disclosure of intellectual property, disruptions to operations and, in the case of machine learning systems acting as industrial con-trol systems, harm to property and persons.

As per other emerging technologies, such as the Internet of Things, machine learning presents a variety of challenges in the management of cyber security risks. In Industry 4.0, material challenges include modelling vulnerabilities of machine learning systems due to the lack of a theoretical basis to rule out attacks, risk mod-elling the complexity and connectivity of systems in Industry 4.0, managing the risk in continually evolving ecosystems and processes in Industry 4.0, and a lack of a cat-alogue of controls to assist in a determination of relevant machine learning controls to apply. With such challenges, how then can the cyber security risk management of a machine learning system in Industry 4.0 be performed?

To this end, we propose a cyber security risk management framework that allows for the management of these machine learning cyber security risks in Industry 4.0. The bases for this risk management framework are a risk model, developed as part of this study, of the machine learning risks and an analysis undertaken of the material challenges risk management faces in Industry 4.0. A selection of cyber security risk methods and techniques is performed and enhancements proposed.

The resulting framework consists of the CORAS risk methodology, further en-hanced to meet these key challenges, an enen-hanced Open Group Standard method-ology on Dependency Modelling, instantiations of HAZOP for machine learning systems, a security assessment mechanism for the adversarial training of machine learning systems, catalogues of machine learning threats and security controls, and a standard of machine learning controls with implementation guidance.

(4)

(5)

Acknowledgements

Thanks for the assistance from the Cyber Security Academy: Jan van der Lubbe, first supervisor, for his patience and especially his helpful advice on machine learning. Dennis de Geus, second supervisor, for his insights into risk management.

As the company that I worked with on the case study remains obfuscated by agreement, I have a challenge in thanking all those, without whose help, this re-search would never have been realised. Nonetheless, their support in all stages of the research, including the conceptualisation and scoping discussions, the execution of the case study and expert reviews, was invaluable.

Sadness in the thanks given to Arjen Kamphuis for his generosity of ideas, guid-ance and encouragement, as he is still missing.

Special thanks to the expert reviewers that I can name for their painstaking con-sideration of the framework, including Pieter Burghouwt, Lecturer, The Hague Uni-versity of Applied Sciences; René Tieben, Principal, Cyber Defense at Capgemini Invent; and not least, Mass Soldal Lund, Associate Professor, Norwegian Defence Cyber Academy, Norwegian Defence University College.

Thanks to Kate Labunets, Postdoctoral Researcher, Technical University Delft for her helpful research suggestions.

Great thanks to the proof readers, Max Denyer-Green, Matthijs Toose, Richard Derks and Dorien Zwaneveld.

(6)

(7)

List of Figures

1.1 The Four Industrial Revolutions, Reprinted From [49, p. 1] . . . 2

1.2 Analysis, Selection, Improvement and Validation Steps . . . 11

2.1 Supervised vs Unsupervised Machine Learning Problems, Reprinted

From [33, p. 1] . . . 15

2.2 Supervised Learning, Reprinted From [106, p. 1] . . . 16

2.3 Venn Diagram, Showing Deep Learning, Reprinted From [41, p. 9] . . 16

2.4 Taxonomy of Attacks of Machine Learning Systems, Divided Over

Three Axes, With Examples, Reprinted From [15, p. 18]. IDS refers

to Intrusion Detection Systems used in cyber security defences. . . 21

2.5 Threat Model Taxonomy, Reprinted From [76, p. 9]. . . 24

2.6 Threat Actor Topology based on CSAN 2016, Reprinted From [23, p. 59] 29

4.1 Hierarchy of Criteria for Artefact Evaluation, Reprinted From [83,

p. 24] . . . 46

5.1 A Taxonomy of Information Security Risk Assessment (ISRA)

Ap-proaches, Reprinted From [95, p. 7] . . . 52

5.2 The Four Quadrants with Risk Classifications, Reprinted From [113,

p. 157]. . . 53

5.3 Summary of Existing Information Security Risk Assessment Approaches,

Reprinted From [95, p. 14] . . . 55

5.4 Typical Workflows of Reactive and Proactive Defensive Mechanisms

Reprinted, Reprinted From [63, p. 7] . . . 61

5.5 An Example Dependency Model for a Trip, Reprinted From [44, p. 19] 63

5.6 ISO 31010 - High-Level View of the Security Risk Management

Pro-cess, Reprinted From [3] . . . 64

5.7 Industry 4.0 Standards, Reprinted From [37] . . . 66

6.1 Risk Management Framework, including Methods/Techniques,

Cat-alogues and Standards . . . 70

6.2 Risks Method Completeness, Reprinted From [113, p. 119] . . . 72

6.3 Modification to Include Knowledge to the Basic Concepts of the CORAS

Method, Adapted From [31, p. 28] . . . 74

6.4 Extract of Key Fields from the Table, including Guide Words for

Ma-chine Learning Systems . . . 77

7.1 Key Evaluation Criteria and Evaluation Methods. . . 88

8.1 Enhancements to the CORAS Steps . . . 91

9.1 Design Science Research Knowledge Contribution Framework, Reprinted

(12)

[31] . . . 109

B.2 CORAS Diagram - Learn Training Data . . . 110

B.3 CORAS Diagram - Learn Machine Learning System . . . 111

B.4 CORAS Diagram -Overload Availability Attack. . . 112

B.5 CORAS Diagram - Loss of Integrity. . . 113

B.6 CORAS Diagram - Running Another Program and Attacking Other Systems. . . 114

B.7 Dependency Analysis Page 1 . . . 116

B.8 Dependency Analysis Page 2 . . . 117

(13)

List of Tables

2.1 Catalogue of Pre-requisites for Attacks and Defences Specific to those

Attacks . . . 26

2.2 Catalogue - Derived Taxonomy for Industry 4.0. . . 28

2.3 Catalogue - Derived Threat Actors for Industry 4.0 . . . 31

2.4 Catalogue of General and Specific Machine Learning Defences . . . 36

5.1 Criteria applied to CORAS, OCTAVE and CRAMM. (Evaluation per the assessment of Shameli-Sendi unless shown.[95]) . . . 58

6.1 Defences Analysis . . . 79

6.2 Risk Management Framework Components Summary - Relevance and CORAS Integration/Usage . . . 82

7.1 Design Science Research Evaluation Strategy Selection Framework, Reprinted From [108, p. 10] . . . 85

7.2 Outcome from Design Science Research Evaluation Strategy Selection Framework, Reprinted From [108, p. 10] . . . 86

B.1 CORAS Risk Acceptance . . . 106

B.2 Company Risk Matrix . . . 107

B.3 Risk Matrix. . . 108

(14)

(15)

Chapter 1

Introduction

1.1 Context and Relevance of Research

Machine learning is a prevalent Artificial Intelligence technique that is being im-plemented in a wide variety of domains. We can see the growing importance of machine learning in industry and in our daily lives. Machine learning is rapidly be-coming an essential building block of applications ranging widely from security to medical diagnosis and treatment, to self-driving cars. These systems are currently being used to tackle previously intractable problems including the reconstruction of brain circuits and the analysis of particle accelerator data. Machine learning has

achieved success in both speech recognition and natural language understanding.[6]

Consumer applications are also being developed, such as Google Assistant which utilises Google Duplex. Google Assistant can hold conversations with a

human-like tone and perform real-world tasks.[11] Western technology firms, including

Al-phabet (Googles parent company), Amazon, Apple, Facebook, IBM and Microsoft are investing heavily to develop their AI capabilities, as are their counterparts in

China.[30]

Elsewhere in the field of cyber security, machine learning techniques are increas-ingly being adopted in cyber security defences, including the detection of

intru-sions,[60] analysing malware[89] or in code analysis and detecting potential exploits

in other programs.

Financial investments in machine learning are increasing: Annual external in-vestment in AI estimated by McKinsey in 2016 was between $8B and $12B. Machine

learning attracted nearly 60 per cent of that investment.[80] Investments are

pre-dicted to increase with worldwide spending on cognitive and artificial intelligence

(AI) systems as a whole forecasted to reach $57.6 billion in 2021.[47]

1.1.1 Machine Learning Applied in Industry

One of these domains where machine learning systems are being applied is in In-dustry 4.0. The term InIn-dustry 4.0 originates from the concept of a fourth industrial revolution, with the introduction of internet technologies into industry. The four

revolutions are shown in Fig.1.1.

In the domain of Industry 4.0, machine learning is being applied increasingly in industrial design, process planning, process modelling and control, automated in-spection, diagnostics and quality control, production planning and control, robotics

and assembly.[68, p. 704]

Some examples in industry are machines that can predict failures and trigger maintenance processes autonomously. Other examples include self-organised

(16)

FIGURE 1.1: The Four Industrial Revolutions, Reprinted From [49, p. 1]

1.1.2 Benefits to Industry

The utopia of Industry 4.0 is analytic technologies being directed at real-time capa-ble and self-organising value chains optimised with respect to values such as costs,

availability, energy and resource consumption, flexibility, and throughput time.[38,

p. 10] In combination with other internet technologies, some of the realisable benefits to industry include:

• "increased visibility of industrial control system activities (work in process,

equipment status, production schedules) and integrated processing systems from the business level, contributing to the improved ability to conduct analy-ses to drive down production costs and improve productivity;

• integrated manufacturing and production systems that have more direct access

to business level information, enabling a more responsive enterprise;

• common interfaces that reduce overall support costs and permit remote

sup-port of production processes;

• remote monitoring of the process control systems that reduces support costs

and allows problems to be solved more quickly."[4, p. 4.2]

These benefits arise because machine learning systems are natural tools at lower levels of the application of intelligence to manufacturing, "where abilities of sensor integration, signal processing, uncertainty handling, real-time and adaptive

func-tioning are required".[68, p. 704]

Other benefits arise because machine learning systems can be used as building blocks in larger information process blocks because of their effective learning

(17)

1.1.3 Susceptibility to Attacks

It has been known for some time that machine learning systems can be tricked and studies have shown that machine learning systems are vulnerable to a space of ad-versarial attacks, including attacks on their confidentiality, integrity and

availabil-ity.[15] In one integrity attack example, researches made tiny changes to a number

of pictures.[5] These pictures were then analysed by a widely used image

recogni-tion system. The researchers found that changing one pixel in about 74 per cent of the test images made the systems incorrectly label what they saw. Such tiny changes can mean that a turtle, for example, is then incorrectly classified as a rifle.

Machine learning systems are also vulnerable to being reprogrammed and used

for nefarious acts. (See Subsection2.3.1for further details.) Recently, there has been

a large influx of contributions to the issue of adversarial attacks posing genuine and severe threats to the promise of applied machine learning. The literature includes the design of adversarial attacks and proposed defences against them. Such attacks are not purely theoretical, as the literature increasingly emphasises that adversar-ial attacks are possible in real-world conditions and hence pose real threats when

incorporated into industry.[6] For example, in one study, specially crafted posters

were effectively used to fool road sign recognition systems in practical drive-by

set-tings.[36, p. 2]

1.1.4 Harm of Attacks

The potential harm to business and broader society from their failure to operate cor-rectly grows with the increasing usage of machine learning systems. Moreover, ma-chine learning systems are increasingly being used independently of external con-trol and influence, as per the pursuit of autonomous (self-driving) cars. Hence, it

becomes increasingly crucial that machine learning systems behave as intended.[92]

In Industry 4.0, the potential impacts of attacks may be equally severe as en-countered elsewhere. The most severe impacts from their application may well be where machine learning systems act as industrial control systems, including control-ling robots and other industrial equipment, as they may induce harm to property and persons. Indeed, concerns over so-called smart autonomous robots are prompt-ing the consideration of ethical questions and the liability for the harm they may

cause.[71]

Other consequences in Industry 4.0 include the disclosure of intellectual property (an adversary can extract intellectual property memorized by the machine learning

system[97]), disruptions to operations, disclosure of the machine learning models

themselves, leading to the loss of competitive advantages gained by using the ma-chine learning systems and many more consequences, depending on the application of machine learning.

The consequences can extend beyond industry itself, because machine learning systems can be reprogrammed and used as a platform to attack other targets.

1.1.5 Risk Management

These cyber security risks then need to be managed in Industry 4.0, which in this context means the "identification, evaluation and prioritisation of risks, followed by coordinated an economical application of resources to minimize, monitor and

(18)

1.1.6 Challenges in Risk Management

As per other emerging technologies, such as the Internet of Things, machine learn-ing presents challenges in managlearn-ing cyber-security risks. As expanded upon and explored further in our research, material challenges in Industry 4.0 include mod-elling vulnerabilities to machine learning systems due to the lack of a theoretical basis to rule out attacks; risk modelling the complexity and connectivity of systems in Industry 4.0; evaluating the risks in the context of unclear legal liabilities for inci-dents; risk modelling in the absence of historical or quantitative data; risk modelling human and machine learning system interactions in industrial processes, including supervision; managing the risk in continually evolving ecosystems and processes in Industry 4.0; managing risks between different stakeholders in the industry value chain; and a lack of an appropriate set of security controls to assist in the determi-nation of machine learning controls to apply. Unfortunately, however, there is little guidance in the literature as to how to apply cyber security risk management to machine learning systems. How then can the cyber security risk management of a machine learning system in Industry 4.0 be performed? We first, however, define the boundaries of our research in the following section.

1.2 Scoping

The scope of this research is limited to the types of systems and the types of learning set out here. The terms used in this scoping section are considered more fully in

Section2.2. Wider ethical, privacy and regulatory implications are excluded from

this research to the extent considered below.

1.2.1 Types of Systems

Unfortunately, the types of systems that are likely subject to adversarial attacks, for good reasons, are not yet well-defined in the literature, with the scope being de-scribed loosely as machine learning systems (but including Bayesian learner and a

support vector machine).[46]

In this research, the scope is limited to machine learning systems implemented using artificial neural networks, because such systems are becoming prevalent and recently there has been a significant influx of contributions in the literature relating

to machine learning systems[6, p. 1][79, p. 1] although attacks on other types of

arti-ficial intelligence systems have been researched. See for example,[20]. Other systems

are excluded because there is less information about such attacks to form a model and because attacks on other machine learning systems may require alternative risk management approaches, especially in terms of differing risk treatments.

1.2.2 Types of Learning

Papernot et al. do not make precise the scope of machine learning, but include so-called supervised learning, unsupervised learning and reinforcement learning, al-though they summarise that work on machine learning security and privacy “to date has for the most part conducted in supervised settings, especially in the context of

classification tasks”.[79, p. 2] However, reinforcement learning system exhibit

(19)

the focus of our enquiry. In summary, this study is limited to supervised and re-inforcement learning in machine learning systems implemented using artificial net-works.

1.2.3 Types of Attacks

Within this scope, confidentiality, integrity, availability and reprogramming attacks are included. Impacts that are accidental in origin, typically considered in the do-main of safety, are excluded.

1.2.4 Risk Management Challenges

There are many challenges in risk management, such as in the optimal allocation of scarce resources to treat the risks as per other cyber security risks. However, the interest and scope of this research is further limited to the aspects of risk manage-ment concerning the identification, evaluation and prioritisation of risks, because these challenges are fundamental in the sense that they must be met before other challenges can be tackled. Also, the focus of this research is on challenges that relate directly to machine learning, rather than general cyber security risk management issues.

1.2.5 Broader Considerations of the Application of Machine Learning

Wider Ethical Considerations

The ethical and legal implications of machine learning being applied in Industry 4.0 and any need for further regulation is beyond the scope of this study. For example, machine learning applications have the potential to replace human decision-makers. At law, this also raises difficult issues of legal liability where the consequence of a

decision results in loss or damage to a third party.[85]

Privacy Considerations

In terms of privacy, it is impossible to separate some privacy aspects from this re-search as “Attacks on confidentiality attempt to expose the model structure or pa-rameters (which may be highly valuable intellectual property) or the data used to

train.”[79] Such data may contain personal data. However, privacy considerations

concerning the general processing of personal data, as regulated under the General

Data Protection Regulation, are excluded.[87]

1.3 The Objectives of the Study

How then can the cyber security risk management of a machine learning system in Industry 4.0 be performed? We propose that risk management can be performed by developing a risk management framework, consisting of risk methodologies, tech-niques and standards.

The goal of this design science research is the development of a cyber security risk management framework that allows for the management of the machine learn-ing cyber security risks in Industry 4.0 and to validate this design artefact within the time and resource constraints available.

(20)

professionals and academics, together with cyber risk practitioners and stakeholders and in particular, those stakeholders in Industry 4.0.

In this paper, the term "risk management framework" (shortened to "the risk framework" or even simply "the framework" or "the artefact") is defined as a com-plementary set of:

• risk methodologies, such as a risk assessment method

• risk techniques, such as a diagram or mapping technique

• catalogues of threats and security controls

• risk standards, such as a set of security controls and guidance

The term risk management framework, as defined, does not include the software or tooling necessary to undertake risk management, as different tools may support the methodologies. Nevertheless, tooling is necessary to give effect to the method-ologies because of the complexity involved, and therefore it is considered in this design science research. Due to time and resource constraints, tool enhancements are simulated rather than developed.

In the absence of a set of machine learning controls to support risk management activities, the drafting of a catalogue of threats and security controls for risk assess-ment and a risk standard, for broader risk manageassess-ment, also forms part of this goal. A design and scoping choice was made not to have the objective of a unified se-curity and safety framework, to address the potential harm to persons and property, but rather to provide a risk management framework that could be integrated with

safety frameworks in industry. (See Section4.3)

In more detail, the objective is that the developed cyber security risk manage-ment framework should:

1. Meet the material machine learning challenges identified in this study. For example, the risk management framework should be able to model risks in light of a lack of a complete theoretical understanding of the machine learning attacks.

2. Satisfy other essential evaluation criteria identified in this study. Primarily, the framework should cover risk management objectives, for example, including all stages of risk management, from context establishment to evaluation and treatment.

The intermediate steps necessary to meet this objective, namely the development of a risk management framework, are then considered in the Research sub-questions and Methodology Sections.

1.4 Research Question

(21)

1.4.1 Sub-questions

The first two research sub-questions are to determine the problem at hand:

• Q1: What are the key cyber security risks in relation to machine learning systems in Industry 4.0?

• Q2: What are the key challenges in undertaking risk management of ma-chine learning systems in Industry 4.0

These are important questions because without knowing the risks and challenges posed, the design of an artefact cannot be successfully undertaken. Here it is essen-tial to identify, describe and analyse both the risks and the risk management chal-lenges, such as, for example, that the threats to machine learning systems are not well understood or precisely defined.

Similarly, it is not sufficient to select methodologies and techniques without first determining what makes for suitable methodologies and the methodologies chosen need to be able to address the challenges posed by machine learning systems. What else makes them suitable? The need to define their suitability leads to the following question about the risk management framework:

• Q3: What are the properties that the risk management framework must sat-isfy?

There are two classes of properties or criteria that the risk management frame-work must satisfy. First, can it perform its risk management function in light of the challenges? Second, there are common artefact evaluation criteria, such as validity, efficacy and completeness that must be met.

With these criteria, existing candidate risk management frameworks can be con-sidered for further development. So the following questions arise:

• Q4: To what extent do any existing candidate risk management frameworks and methods meet the criteria?

• Q5: How can the considered candidates be further improved vis-a-vis the evaluation criteria?

Once these questions are answered and the risk management framework improved, this design research artefact needs to be evaluated.

• Q6: How can the risk management framework be evaluated?

Lastly, the analysis and evaluation methods chosen lead to:

• Q7: What are the limitations of the risk management framework?

These questions form the basis for the methodology and the outline below.

1.5 Prior Literature Review

To the end of achieving the objective of a risk management framework, a review of the literature reveals many shortcomings. These shortcomings are classified in

ac-cordance with Gregor et al.[42, p. 343] The two classes of knowledge per Gregor et

(22)

1.5.1 Descriptive Knowledge

"Descriptive knowledge is the ’what’ knowledge about natural phenomena and the

laws and regularities among phenomena."[42, p. 343] In the domain of machine

learning, there is emerging knowledge about attacks on machine learning systems and classifications or taxonomies of those attacks, knowledge of so-called availabil-ity attacks, and attacks possible in the training phase of machine learning. (See

Chapter2.) Also within the category of descriptive knowledge, attempts have been

made recently in this emerging domain to make sense of and understand those

at-tacks with mixed success.[52, p. 4]

However, there is other descriptive knowledge lacking in the literature that is important for the risk management framework and developed in this research:

First, although there exists a catalogue of machine learning attacks,[63, p. 12]

this catalogue does not include several important elements for risk management, such as:

• Threat actors: Who might attack in practice?

• Pre-requisites for attacks, to understand how to prevent attacks occurring

• Ease of implementation of controls

• Traditional IT controls which help prevent machine learning attacks, such as

network controls

No other catalogues of machine learning attacks are known.

Also, as we will see, there is no standard nor set of machine learning cyber se-curity controls that also aids the implementation of relevant and effective controls.

(See Section5.4.)

Moreover, with respect to the taxonomy of attacks, it has been necessary to cor-rect misnomers in existing taxonomies for the construction of the catalogues in the

risk management framework. (See Subsection2.3.1.)

Lastly, the descriptive knowledge on machine learning risks has not been con-sidered in the context of Industry 4.0, nor reduced to a catalogue or catalogues of threats and controls that can be used in risk management, including performing risk assessments. These are both important in order to understand which of the machine learning attacks are relevant to industry and for there to be information in a practical format that can be used in risk assessments.

1.5.2 Prescriptive Knowledge

"Prescriptive knowledge is the how knowledge of human-built artefacts."[42, p. 243]

It includes constructs, models, methods, instantiations and design theory.

(23)

If we search more broadly outside of machine learning systems, we do find arte-facts for risk management of similar systems in Industry 4.0, in relation to both cy-ber security and safety. For example, some risk methods relating to industrial con-trol systems, which is a similar type of application in Industry 4.0, considered by

Kriaa.[56]

Nevertheless, there is a plethora of cyber security risk assessment methods and techniques, such as CRAMM. Shameli-Sendi et al., have, for example, surveyed risk

assessment methods with various scopes and properties.[95] These are intended to

be of general application in cyber security. To the authors’ knowledge and a re-view of the literature, these artefacts have not been applied to machine learning. However, there are significant challenges in applying these artefacts that need to be selected, extended or refined so that they can be used in the domain of machine

learning and Industry 4.0. (See Chapter3.) In conclusion, to the extent above, the

prior literature does not provide for:

• Sufficient catalogues of machine learning attacks

• Standards of machine learning controls

• A specific risk management framework for machine learning in Industry 4.0

Moreover, the challenges of applying general risk methods and techniques mo-tivate us to select, extend and refine these, in combination with the supporting cata-logues and standards to manage machine learning risks in Industry 4.0.

1.6 Methodology

The overall methodology in this paper is design science research, as the objective of the study is to provide a designed, developed risk management framework for machine learning in Industry 4.0. Design science includes the creation and

evalu-ation of artefacts intended to solve identified organisevalu-ational problems.[109, p. 77]

Both the assessment and the refinement of methodologies are key phases of design

science.[109, p. 80]

The Seven Guidelines for Design Research in Information Systems Research[109,

p. 82] are adopted as the predominant guidance for design science research and the guidelines can be used to increase the rigour of this research. However, these high-level guidelines require further elucidation, and hence the evaluation methodology and design criteria selection methodologies are considered in more detail in Chapter

7and Subsection4.1respectively.

Also, as the contribution to knowledge is seen as the foremost criterion for re-search and publication, the design science knowledge contribution framework and associated guidance of Gregor et al. in relation to design science research is adopted to make clear the contributions in this research and place them in context for

read-ers.[42, p. 338]

1.6.1 Outline

We begin in Chapter2by introducing Industry 4.0, machine learning systems and

(24)

risk management to machine learning in Industry 4.0. Following on in Chapter4, we determine the criteria for applying risk methodologies, before identifying risk management frameworks and risk methodologies. For example, the criteria of the efficacy of applying the methodologies as well as the completeness of coverage of the risk management steps. The literature on design science evaluation and upon the evaluation of risk management methodologies are utilised to form the criteria.

Having an established set of criteria, in Chapter5 we consider and select from

the prime risk methodology candidates available, by means of a literature review.

Then in Chapter6, we consider whether and how they can be adapted to meet the

problem statement and the criteria, and illustrate those adaptations.

The Evaluation Methodology is determined in Chapter7and includes evaluation

by way of a case study and expert review. The results of the evaluation, including

a discussion of the limitations is presented in Chapter8. The limitations of the risk

management framework from several perspectives are considered, as well as future

work. In the final chapter, Chapter9, we present the design science contributions

and conclusions.

The key steps are summarised diagrammatically in Figure 1.2. However, only

(25)

Assets Consequences Likelihood Threat Actors Threats Treatments Vulnerabilities

Model of Risks for Ind 4.0 ML Challenges RM Conditions RM Criteria

Selection Criteria

RM / Framework Candidates

Selected RM Candidates

Instantiated/ Improved RM

Validation Case Study Expert Review

Design Cycle

(26)

(27)

Chapter 2

Industry 4.0, Machine Learning

Systems and their Susceptibility to

Attacks

What is Industry 4.0 and what role do machine learning systems play within Indus-try 4.0? In this chapter, we introduce and define IndusIndus-try 4.0. We also provide a conceptualisation of machine learning systems, including their learning life-cycle, as a basis for understanding the cyber security risks posed to them. To form a model of the risks in Industry 4.0, the risks are then considered from standard perspectives for security and risk management, namely in terms of their assets, consequences, parties, likelihood, defences, threat actors and vulnerabilities. See for example these

perspectives in ISO/IEC 13335-1:2004[1].

This leads to the development in this chapter of the following catalogues of threats and controls artefacts:

• Catalogue of Derived Taxonomy of Attacks for Industry 4.0

• Catalogue of Derived Threat Actors for Industry 4.0

• Catalogue of Pre-requisites for Attacks and Defences Specific to those Attacks

• Catalogue of General and Specific Machine Learning Defences

Together, these catalogues support the usage of the methodologies and tech-niques, such as CORAS, selected and developed in subsequent chapters.

2.1 Industry 4.0

Industry 4.0, refers to the fourth industrial revolution with the introduction of

inter-net technologies into industry.[54, p. 57] See Figure1.1. The term originates from

a project as part of the strategy of the German Government, but has since been

adopted widely.[48, p. 1]

The necessary technological components for Industry 4.0 include IoT, Big Data, cloud computing and artificial intelligence technologies that are likely to include

machine learning systems.[111, p. 216]

While many definitions of the term Industry 4.0 have been formulated, some with marketing hype, one narrow definition of the term Industry 4.0 is the "applica-tion of the generic concept of cyber-physical systems (CPS) to industrial produc"applica-tion

(28)

defined as ‘integrations of computation and physical processes. Embedded comput-ers and networks monitor and control the physical processes, usually with feedback

loops where physical processes affect computations and vice versa.”[110, p. 1]

Another, broader definition is that Industry 4.0 relates to “the brisk transforma-tion in the design, manufacture, operatransforma-tion and service of manufacturing systems and products, where digital technology and the Internet merge together with the conventional industry, achieving digitally connected manufacturing operations with

a highly integrated value chain.”[69] It is this definition that is adopted in this paper,

as the limitation to cyber-physical systems does not encompass the wide range of ways that machine learning systems can be applied to industry, such as the example provided of process planning. Also, in support of adopting the broader definition in this research is the realisation that is possible to have multiple applications or use cases of machine learning systems in one factory, and hence adopting a narrow definition could mean that the risk management would need to be reconsidered for say, cyber-physical systems and service maintenance in different risk management frameworks.

The utopia of Industry 4.0 is that in a smart factory, machines and products com-municate and negotiate with each other, to reconfigure themselves for flexible pro-duction. In order to do this, there needs to be massive data collection, analysis

to optimise system performance.[111, p. 216] However, one could argue that such

a utopia of interconnectedness may take some years to realise, especially as more piecemeal and selective investment approaches to integrate internet technologies may have more compelling business cases. Nevertheless, wherever and however machine learning systems are introduced to industry, the need for risk management remains because of the potentially severe consequences.

2.2 Machine Learning, Artificial Neural Networks and

Learn-ing Methods

What is meant by machine learning and how is it realised in practice? We consider in this section the learning algorithm, the various types of machine learning, and the machine learning system life cycle so as to have a firm base for considering the risks to machine learning.

In Artificial Intelligence, machine learning methods are techniques that allow computer systems to improve with experience and data, and a machine learning

algorithm is an algorithm that is able to learn from data.[41, p. 99]

2.2.1 Supervised, Reinforced and Unsupervised Learning

Machine learning can be divided into supervised and unsupervised learning. See

Figure2.1. In terms of differences between the two types of learning, the term

su-pervised learning originates from the view of the target or label being provided by a

trainer who shows the machine learning system what to do,[41, p. 105] often by

man-ually annotating the label to form an encoded pair (input, output classification).[62]

In contrast, in unsupervised learning, patterns in the input are learnt, even though no explicit feedback or labelling is supplied, such as in the example of the

learn-ing task of clusterlearn-ing.[93, p. 264] In practice, the distinction between supervised and

unsupervised can be blurred. For example, in semi-supervised learning, some

(29)

FIGURE 2.1: Supervised vs Unsupervised Machine Learning

Prob-lems, Reprinted From [33, p. 1]

Reinforcement learning, whereby the machine learning system learns from a se-ries of reinforcements, with rewards and punishments and there is a feedback loop between the learning system and its experiences, is very close in terms of technique

and in terms of often being realised in neural networks.[41, p. 106] As we shall see in

Subsection2.3.1, similar susceptibilities to attacks exist as per supervised learning,

and hence reinforcement learning is included.

2.2.2 Hypothesis Building of Supervised Learning - Classification and

Regression Tasks

Within supervised learning, there are two types of tasks: Classification tasks produce a discrete output from a set of finite alternatives whereas regression tasks produce

a continuous output.[115, p. 7] Then in more detail, Wongrassamee et al. provide a

brief explanation that "a learning algorithm of a system takes in a set of training data and obtain a hypothesis function (h). This function is used in the training phase to map the input features values to the answer of either a classification or regression

problem.”[115, p. 8] Although a simplification, “it can be thought that the optimal

hypothesis function for a regression task is the best fit line for the data points ... and the hypothesis function of a classification task would be the best line that separate

between the different classes of data points.”[115, p. 8]. The breakdown of machine

learning into supervised and unsupervised learning and further into clustering,

clas-sification and regression is shown in Figures2.1and2.2.

2.2.3 Deep Learning

One particular form of machine learning, deep learning (also known as deep

struc-tured learning or hierarchical learning) is being widely adopted.[41, p. 167] Deep

learning is "a particular kind of machine learning that achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract

representa-tions computed in terms of less abstract ones.”[41] The relationship between

artifi-cial intelligence, machine learning, deep learning and representative learning can be

seen in Figure2.3. Hence, machine learning includes deep learning.

(30)

FIGURE2.2: Supervised Learning, Reprinted From [106, p. 1]

FIGURE 2.3: Venn Diagram, Showing Deep Learning, Reprinted

(31)

(MLP) or feed-forward neural networks.[41, p. 168] They are computer implementa-tions of “large neural networks organized into layers of neurons, corresponding to

successive representations of the input data.”[78, p. 2] MLPs are the quintessential

deep learning models.[41, p. 168]

2.2.4 The System Life Cycle of Machine Learning Systems

For machine learning to occur, the following phases are typical. Here, a narrow definition of the system life cycle, limited to data processing, learning and execution, is used in the paper as the focus is on cyber security risks, not project risks. A broader project management life cycle would typically include activities such as identifying potential applications for machine learning that will generate business value.

Measurement Phase

In the first phase of measurement, real-world objects need to be transformed into amenable representations that machine learning systems can process, often remov-ing extraneous elements that are irrelevant to the learnremov-ing. Each feature of a mea-surement is usually a real, an integer, a boolean or a category. For supervised learn-ing, a dataset contains features, but each example is also associated with a label or

target.[41, p. 105] The dataset arrived at represents observations of the environment

and served as the basis for learners ability to learn and predict.

Feature Selection Phase

After a dataset is collected, it is often refined by selecting its features aspects of the

data most relevant to the learning task.[70, p. 28] This task is often repeated with the

original data set to find an optimal set of features.

Model Selection Phase

Finding the best model is often a case of trial and error. Russell et al. state that learning is “a search through the space of possible hypotheses for one that will

per-form well.”[93, p. 695] This phase involves measuring and attempting to improve

the accuracy of a hypothesis.

Training Phase

The process of training can be by way of batch training, where the machine learning system trains on a training set and is evaluated on an evaluation set. Alternatively, the machine learning system repeatedly trains online, where the machine learning

system continually obtains its labels.[70, p. 29] For the reader, a rigorous treatment of

the training of machine learning systems, including explanations of loss functions, gradient descent, forward propagation and backward propagation can in a book by

Goodfellow et al.[41]

Data in the Training Phase In this training phase, in fact three different training

sets are commonly used and can be defined as the following:[103][41, pp. 104, 120]

(32)

• Validation Dataset: The sample of data used to provide an unbiased evalua-tion of a model fit on the training dataset while tuning model hyperparame-ters.

• Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

These training phase datasets are created by splitting the input data used in the training phase. However, because they are split and are in practice processed in the training phase in nearly the same manner, albeit for different purposes, then for the remainder of this paper, they are referred to collectively as the training data.

Inference/Evaluation/Test Phase

Once trained on a dataset, the learned hypothesis of the machine learning system is

then used to predict the response variables or labels for a set of unlabelled data.[70,

p. 31] Many different terms are used for this phase in the literature. (See for example

[115, p. 7] and [70, p. 31].) The term evaluation phase is used in this paper for this

phase.

Data in the Evaluation Phase The data upon which the evaluation is performed is referred to in this paper as the evaluation data. There are some life-cycle differences that exist with reinforcement learning, through the phases of training and evaluation

can be observed in reinforcement learning as well. (See [8].) In practice, the life cycle

phases are iterated, particularly in the context of Industry 4.0 and ever-changing circumstances and continual improvements.

2.2.5 Machine Learning Systems Can Be Fooled

Due to limitations or imperfections with the training of machine learning systems,

they can be fooled and compromised.[76, p. iii] For example, with regard to the

integrity of machine learning systems, they are vulnerable to so-called adversar-ial inputs that can cause the algorithms to fail in the primary objective of correctly

classifying those inputs.[78, p. 1] That is, “given a correctly-classified input x, it is

possible to find a new input x that is very similar to x but is assigned a different

la-bel."[102, p. 1] This vulnerability allows, for example, an attacker to seize control of

autonomous vehicles to make wrong decisions on recognizing traffic signs and voice

control systems to recognise false voice commands.[63] Similar to attacks on their

in-tegrity are attacks on machine learning systems which can cause a denial of service

such that the machine learning system is rendered unavailable in practice.[70, p. 38]

As well, in relation to the confidentiality of machine learning systems, attacks are possible whereby access to the machine learning system allows for the "backing out" of sensitive data that has been used to train the system. Frederikson et al. pro-vide an example of a facial recognition attack, whereby an attacker can reproduce a recognizable image of a person, with only access to an application program interface (API) to the machine learning system that has learnt using the facial images, and the

name of the person whose face is recognized by it.[39, p. 1]

(33)

lead to their re-purposing as “spies or spam bots” or, in the case of recurrent neural

networks, mining crypto-currency.[35, p. 8]

Why can machine learning systems be fooled? In relation to integrity attacks: “While numerous hypotheses compete to provide an explanation for adversarial

samples, their root cause still remains largely unknown.”[119, p. 46] In fact, from

another point of view, such susceptibilities are not surprising, since machine learn-ing models constructed "only work on a very small amount of all the many possible

inputs they might encounter."[52, p. 4]

Furthermore, adversarial examples have also been found to transfer across mod-els trained with the same machine learning technique, but also on modmod-els not in-volving neural networks, and even on ensembles taking collective decisions that

may consist of various machine learning techniques, including neural networks.[75,

p. 4]

Research into their real-world applicability of the susceptibilities is limited, but there are strong suggestions that these systems can be exploited in practice. For example, adversarial images can be used to fool a system even after being printed

and recaptured with a cell phone camera.[72, p. 8]

Concerning why confidentiality can be breached, neural networks have the

ca-pacity to remember everything they have been trained upon,[120, p. 2] which then

allows for the training data to be extracted with a carefully crafted algorithm.

2.3 Risk Management

Knowing the potential for machine learning systems to be fooled, we then analyse at a high level the possible threats and the consequences of such attacks in the context of Industry 4.0. How can we analyse the risks in a structured way?

Common components of risks in cyber security have been identified. Wangen identifies the common components of asset evaluation, threat, vulnerability and con-trol assessments components of risk assessment as being at the core of most risk

management frameworks.[113, p. 3] One of the frameworks reviewed, CORAS, is

used here to provide a taxonomy of risk components and their definitions, although at this stage of the analysis, other risk frameworks that are based on a good risk con-ceptualisation, such as those surveyed by Wangen, could have been used instead of

CORAS.[113]

CORAS provides the following definition of risk terms:

• Asset: “Something to which a party assigns value and hence for which the

party requires protection”

• Consequence: “The impact of an unwanted incident on an asset in terms of

harm or reduced asset value”

• Likelihood: “The frequency or probability of something to occur"

• Party: “An organization, company, person, group or other body on whose

behalf a risk analysis is conducted”

• Risk: “The likelihood of an unwanted incident and its consequence for a

spe-cific asset”

(34)

• Threat Scenario: “A chain or series of events that is initiated by a threat and that may lead to an unwanted incident”

• Treatment: “An appropriate measure to reduce risk level”

• Unwanted Incident: “An event that harms or reduces the value of an asset”

• Vulnerability: “A weakness, flaw or deficiency that opens for, or any be

ex-ploited by, a threat to cause harm to or reduce the value of an asset”

Of these, only asset, consequence, party, threat (including threat actors) and treat-ment (also referred to as a defence here, although treattreat-ments can include post inci-dent measures, such as the restoration of a system) can be considered generically for Industry 4.0, as the actual risks depend on the specific implementation of the machine learning system in industry. The remainder are left for risk assessments in practice, conducted on a particular factory or value chain. However, threats are fur-ther divided into threats and threat actors, as ofur-therwise the analysis of likely hackers would be excluded.

The objective in this subjection is to identify for each component, any limits in their application to Industry 4.0 (For example, are there any machine learning threats not likely to be manifested in Industry 4.0?).

We start with the threats and the attacks on machine learning systems themselves and the taxonomies available to classify machine learning attacks, so as to have a structure and a language for the consideration of the risks.

2.3.1 Threats and Taxonomies of Attacks

Different taxonomy of attacks that classify the machine learning attacks have been formulated. However, the taxonomies are incomplete and can be misleading.

First, a taxonomy that describes the space of attacks against neural networks has

been presented by Barreno.[15, p. 2] See Figure2.4. In the figure, per the taxonomy

of Barrenno, there are then three axes of:

1. the capability of the attacker

2. the type of security violation the attacker causes

3. the attacker’s intention

This taxonomy has been extended by Huang et al. to include privacy violations,

which they state are qualitatively different to integrity and availability violations.[46,

p. 44]

Huang et al. describe the first axis as “the capability of the attacker: whether (a) the attacker has the ability to influence the training data that is used to construct the classifier (a causative attack) or (b) the attacker does not influence the learned classifier, but can send new instances to the classifier and possibly observe its

de-cisions on these carefully crafted instances (an exploratory attack).”[46, p. 44] Both

the labelling and definition of exploratory attacks is misleading as it suggests that it is only about observation and exploration. Rather, Barreno also uses "exploratory" more generally to cover any attacks that "take place post training" and "exploratory

attacks exploit misclassifications but not affect training".[15, p. 4][15, p. 15] It is this

(35)

2.3. Risk Management 21

Causative:

Targeted

Indiscriminate

Exploratory:

Targeted

Integrity

The intrusion foretold: mis-train a

particular intrusion

The intrusion foretold: mis-train

any of several intrusions

Availability

The rogue IDS: mis-train IDS to

block certain traffic

The rogue IDS: mis-train IDS to

broadly block traffic

Indiscriminate

The shifty intruder: obfuscate a

chosen intrusion

The shifty intruder: obfuscate any

intrusion

The mistaken identity: censor a

particular host

The mistaken identity: interfere

with traffic generally

Table 2.1: Our taxonomy of attacks against machine learning systems, with examples

from Section 2.2.3.

F

IGURE2.4: Taxonomy of Attacks of Machine Learning Systems,

Di-vided Over Three Axes, With Examples, Reprinted From [15, p. 18].

IDS refers to Intrusion Detection Systems used in cyber security de-fences.

beyond only attacks in the training and evaluation phases, as Nelson describes at-tacks which are possible at every phase of the system life-cycle, including before

training.[70, p. 28]

This axis also then is misleading when described as the "capability of the at-tacker", as it is not really concerned with the capability of the attacker when given its usual meaning, such as the attacker’s competence or resources to mount an at-tack, but rather timing of the attack in the machine learning life cycle.

The second axis in the taxonomy according to Huang et al. concerns the type of security violation the attacker causes: either an integrity violation (generating false negatives), an availability violation (generating so many false negatives or false positives, that the system becomes effectively unusable) or a privacy violation,

al-lowing the inference of confidential information used in the learning process.[46,

p. 44] Huang et al. include the secrecy of the model in the term “Privacy”, which would seem to include extraction of model and data attacks in general, regardless of whether privacy (of individuals) is violated. However, in this paper, the term pri-vacy attacks are used to refer to attacks to yield private data about individuals, often referred to as personal data, and confidentiality attacks as a wider term, to yield con-fidential data (which may well include private data about individuals). These types

of attacks echo the so-called trinity "CIA definition" dominant in cyber security.[53]

(36)

of consistent behaviour and results", defined in the standard is difficult to distin-guish from the property of "integrity - property of accuracy and completeness" as an attack on the integrity of a machine learning system is likely to affect both the accuracy and consistency of behaviour and results.

Missing from the definition of integrity by Huang et al. are false positives. Also missing from the definition of availability are volume attacks (not necessarily false) that may use training or other data to simply overload a system. Such brute force availability attacks would require less adversarial manipulation of data.

The third axis then refers to how specific the attackers intention is: “whether (a) the attack is highly targeted to degrade the classifier’s performance on one particu-lar instance or (b) the attack aims to cause the classifier to fail in an indiscriminate fashion on a broad class of instances.” Similarly, the definition is perhaps unfortu-nately described in this way, as it is the broadness of the attack that is the essence of the definition and the axis, and not the intention per se of the attacker which can be used to classify the attack. Nevertheless, each of the three axes can potentially be a spectrum of choices, rather than a binary classification.

Two new types of attacks require further consideration as to whether they fit the attack taxonomy of Huang et al. First, a new form of adversarial attack has been identified, a so-called reprogramming attack. As foreshadowed previously, in such an attack, the neural network is reprogrammed to perform a novel task chosen by the attacker. As the outcome of the attack, namely a reprogramming of the neural network, it should be an additional class of attacks. It is not yet clear from the re-search literature whether the neural network can be reprogrammed to perform an additional function and yet, for the most part, continue to provide its original clas-sification function.

Second, attacks on reinforcement learning systems may include an attack called

a policy induction attack.[16, p. 262] However, the attack can be thought of a

re-peated integrity attack according to the Behzadan et al., and so falls within the

tax-onomy.[17, p. 268]

We therefore have the following definitions:

• Integrity Attack: An attack generating false negativesand false positives

• Availability Attack: An attack generating so many false negatives or false

pos-itives, that the system becomes effectively unusable

• Confidentiality Attacks: An attack allowing the inference of confidential

in-formation (which may or may not include private inin-formation) used in the learning process

• Reprogramming Attack: An attack where the machine learning system (also

known as classifier) is reprogrammed to perform a novel task chosen by the attacker

• Causative Attack: The attacker has the ability to influence the training data

that is used to construct the machine learning system

• Exploratory Attack: The attacker does not influence the learned classifier, but

can send new instances to the machine learning system and possibly observe its decisions on these carefully crafted instances

• Targeted Attack: The attack is highly targeted to degrade the classifier’s

(37)

• Indiscriminate Attack: The attack aims to cause the classifier to fail in an indis-criminate fashion on a broad class of instances

With these modifications and clarifications of the taxonomy of Huang et al. and the additional reprogramming attacks, we nevertheless have a workable taxonomy of attacks as a basis for further analysis in Industry 4.0.

Further Refinement of the Taxonomy

A further breakdown of the taxonomy, but only for integrity attacks, is proposed by

Papernot.[76, p. 9] Here, Papernot classifies the types of adversarial goals.

These are:

• Confidence reduction- reduce the output confidence classification

• Misclassification- alter the output classification to any class different from the original class

• Targeted misclassification- produce inputs that force output classification into a specific target class

• Source/target misclassification - force the output classification of a specific input to be a specific target class

As well, Papernot provides a further breakdown by the information and capabil-ities at an attacker’s disposal:

1. Training data and network architecture - This adversary has perfect knowl-edge of the DNN used for classification.

2. Network architecture - This adversary has knowledge of the network archi-tecture F and its parameter values.

3. Training data- This adversary is able to collect a surrogate dataset, sampled from the same distribution as the original dataset used to train the DNN.

4. Oracle- This adversary has the ability to use the neural network (or a proxy of it) as an oracle.

5. Samples- This adversary has the ability to collect pairs of input and output related to the neural network classifier.

These categories of information in Fig. 2.5 are not fixed, with Xiao et al. also

adding gradations in knowledge that include feature representation (See Phases in

Subsection2.2.4) and knowledge of the feature selection algorithm.[116, p. 3]

Never-theless, the taxonomies allow for a structure to reason about the security and privacy

of systems that incorporate machine learning.[79, p. 2]

Although Papernot stresses that “the integrity of the classification is of paramount importance’, this capability axis of Papernot could be also be used to further classify availability and privacy attacks from the extended taxonomy of Huang. Similarly, the availability and privacy attacks may well be broken down further into adversar-ial goals in further research for completeness of the consideration and classification of attacks.

(38)

The Design of a Risk Management Framework for Machine Learning Systems in Industry 4.0