Machine Learning Systems Can Be Fooled

Due to limitations or imperfections with the training of machine learning systems,

they can be fooled and compromised.[76, p. iii] For example, with regard to the

integrity of machine learning systems, they are vulnerable to so-called adversar- ial inputs that can cause the algorithms to fail in the primary objective of correctly

classifying those inputs.[78, p. 1] That is, “given a correctly-classified input x, it is

possible to find a new input x that is very similar to x but is assigned a different la-

bel."[102, p. 1] This vulnerability allows, for example, an attacker to seize control of

autonomous vehicles to make wrong decisions on recognizing traffic signs and voice

control systems to recognise false voice commands.[63] Similar to attacks on their in-

tegrity are attacks on machine learning systems which can cause a denial of service

such that the machine learning system is rendered unavailable in practice.[70, p. 38]

As well, in relation to the confidentiality of machine learning systems, attacks are possible whereby access to the machine learning system allows for the "backing out" of sensitive data that has been used to train the system. Frederikson et al. pro- vide an example of a facial recognition attack, whereby an attacker can reproduce a recognizable image of a person, with only access to an application program interface (API) to the machine learning system that has learnt using the facial images, and the

name of the person whose face is recognized by it.[39, p. 1]

Additionally, as well as several researched instances of confidentiality, integrity and availability attacks, machine learning systems can also be subjected to a so- called reprogramming attack. In such an attack, the neural network is reprogrammed to perform a novel task chosen by the attacker. These re-programming attacks could

lead to their re-purposing as “spies or spam bots” or, in the case of recurrent neural

networks, mining crypto-currency.[35, p. 8]

Why can machine learning systems be fooled? In relation to integrity attacks: “While numerous hypotheses compete to provide an explanation for adversarial

samples, their root cause still remains largely unknown.”[119, p. 46] In fact, from

another point of view, such susceptibilities are not surprising, since machine learn- ing models constructed "only work on a very small amount of all the many possible

inputs they might encounter."[52, p. 4]

Furthermore, adversarial examples have also been found to transfer across mod- els trained with the same machine learning technique, but also on models not in- volving neural networks, and even on ensembles taking collective decisions that

may consist of various machine learning techniques, including neural networks.[75,

p. 4]

Research into their real-world applicability of the susceptibilities is limited, but there are strong suggestions that these systems can be exploited in practice. For example, adversarial images can be used to fool a system even after being printed

and recaptured with a cell phone camera.[72, p. 8]

Concerning why confidentiality can be breached, neural networks have the ca-

pacity to remember everything they have been trained upon,[120, p. 2] which then

allows for the training data to be extracted with a carefully crafted algorithm.

2.3

Risk Management

Knowing the potential for machine learning systems to be fooled, we then analyse at a high level the possible threats and the consequences of such attacks in the context of Industry 4.0. How can we analyse the risks in a structured way?

Common components of risks in cyber security have been identified. Wangen identifies the common components of asset evaluation, threat, vulnerability and con- trol assessments components of risk assessment as being at the core of most risk

management frameworks.[113, p. 3] One of the frameworks reviewed, CORAS, is

used here to provide a taxonomy of risk components and their definitions, although at this stage of the analysis, other risk frameworks that are based on a good risk con- ceptualisation, such as those surveyed by Wangen, could have been used instead of

CORAS.[113]

CORAS provides the following definition of risk terms:

• Asset: “Something to which a party assigns value and hence for which the

party requires protection”

• Consequence: “The impact of an unwanted incident on an asset in terms of

harm or reduced asset value”

• Likelihood: “The frequency or probability of something to occur"

• Party: “An organization, company, person, group or other body on whose

behalf a risk analysis is conducted”

• Risk: “The likelihood of an unwanted incident and its consequence for a spe-

cific asset”

• Threat Scenario: “A chain or series of events that is initiated by a threat and that may lead to an unwanted incident”

• Treatment: “An appropriate measure to reduce risk level”

• Unwanted Incident: “An event that harms or reduces the value of an asset”

• Vulnerability: “A weakness, flaw or deficiency that opens for, or any be ex-

ploited by, a threat to cause harm to or reduce the value of an asset”

Of these, only asset, consequence, party, threat (including threat actors) and treat- ment (also referred to as a defence here, although treatments can include post inci- dent measures, such as the restoration of a system) can be considered generically for Industry 4.0, as the actual risks depend on the specific implementation of the machine learning system in industry. The remainder are left for risk assessments in practice, conducted on a particular factory or value chain. However, threats are fur- ther divided into threats and threat actors, as otherwise the analysis of likely hackers would be excluded.

The objective in this subjection is to identify for each component, any limits in their application to Industry 4.0 (For example, are there any machine learning threats not likely to be manifested in Industry 4.0?).

We start with the threats and the attacks on machine learning systems themselves and the taxonomies available to classify machine learning attacks, so as to have a structure and a language for the consideration of the risks.