To determine the evaluation methodology, the Framework for Evaluation of Design
Science Research by Venable et al. is adopted here.[108, p. 12] (See Table7.1.) It is
adopted as it presents a systematic method for determining an appropriate evalu- ation methodology in a four-step process and other literature on design science do
not provide such detailed guidance. (See for example, [109, p. 82].)
Following the method of Venable et al., we have identified the methodologies that will be both instantiated and modified, namely the Risk Management Frame-
work, the objective of this study.(Step 1a.)
Also according to the method, this framework socio-technical rather than purely technical as it requires human involvement. Upon an initial examination, the frame- work may be safety critical since the machine learning systems are applied in man-
ufacturing.(Step 1b.)(See Section4.3safety is considered in further detail.)
Considering the properties that need to be evaluated according to their method, we can expect the utility, effectiveness, efficiency and efficacy are appropriate criteria
for evaluation. Ethical considerations are not the focus of this research.(Step 1c.)
Because there is a large number of potential methodologies that could be applied, then, following their step method, it is important to consider which methodologies
have these properties.Step (1d.)Also, side effects should be considered, since safety
is a concern.
Regarding considering the resources as recommended, there are real limits on the research including a limited time frame for the research and limits on budgets
and resources for the evaluation.(Step 1e.)
Matching the conclusions drawn inStep 1., to the author’s framework criteria,
this leads to the result that the evaluation should be Ex Post and Naturalistic.(Step
2.) A valid approach according to the authors for this resulting quadrant, and the
central approach adopted in this research then is a case study.(Step 3)(See Fig.7.2.)
However, due to the inherent limitations of a case study, another form of validation
suggested by Venable et al. is used, namely, expert review. See [108, p. 14] and also
considered by [88].
We first consider the design of the case study itself.[108]
7.2.1 Case Study Design - Evaluation Requirements
Purpose of the Evaluation Possible purposes, per Venable et al., of the evaluation
include:[108, p. 426]
1. "Evaluate an instantiation of a designed artefact to establish its utility and
efficacy (or lack thereof) for achieving its stated purpose
2. Evaluate the formalized knowledge about a designed artefacts utility for achiev- ing its purpose
3. Evaluate a designed artefact or formalized knowledge about it in comparison to other designed artefacts ability to achieve a similar purpose
4. Evaluate a designed artefact or formalized knowledge about it for side ef- fects or undesirable consequences of its use
5. Evaluate a designed artefact formatively to identify weaknesses and areas of improvement for an artefact under development."
TABLE 7.1: Design Science Research Evaluation Strategy Selection
TABLE7.2: Outcome from Design Science Research Evaluation Strat-
egy Selection Framework, Reprinted From [108, p. 10]
For this case study, purpose 1) and purpose 4) are relevant. Purpose 2) is ex- cluded as there is no formalized knowledge about the frameworks. Nor is there another artefact (purpose 3)) to compare. Lastly, while further improvements can be identified, the methodologies are complete, hence purpose 5) is not applicable.
Constraints Constraints on the case study are a limited time frame for the research and limits on budgets and resources for the evaluation. A major limitation in this research is the inability of a case study to assess risk management over a period of time. This limitation means that the challenges of continuous change in machine learning systems and in Industry 4.0 cannot be validated in a single point in time case study, and that alternatives of expert review and rigour in argumentation are necessary. The expert review methodology is considered in the following section.
7.2.2 Expert Review
Due to the limitations of the case study in evaluating all of the parts of the risk frame- work against the evaluands identified, we choose expert reviews of the framework. Even with a panel of reviewers, not all properties can be reviewed in depth. There- fore, the choice is made to focus on the less subjective properties, such as efficacy and completeness, rather than ease of use and fit with the organisation. The fact that the risk management framework could be evaluated as not the easiest to use, and for example, requires training or familiarisation to work well, means that these issues can be remedied with appropriate time, attention and resources in the target organisation and are hence less important. Thus, these properties are not the focus of the expert review. Hence, the suitable evaluation methods are a case study, expert reviews and informed argument.
Expert Review Questions Experts were asked to review in the following manner:
1. Chapters of this paper relevant to the expertise, such as machine learning or risk management and provide general feedback
2. Asked to look at specific artefacts and provide their feedback with respect to particular properties of the artefact
An example extract of the specific request in 2) is shown below:
"Please would you provide feedback with respect to the following:
1. Pre-requisites for Attacks, Table2.1.
2. General and Specific Machine Learning Defences2.4.
3. Conclusions on Risk2.3.12.
Ideally, if you could comment on the following points, with respect to the above items:
• Whether they are fit for purpose(efficacy)
• Whether they are valid, follow a sound method(validity)
• Whether they are general(efficacy)"
Expert Reviewers Due to the sensitivity of the risk assessment, any information identifying the company used in the Case Study ("the Company") involved has been obfuscated.
• Pieter Burghouwt, Lecturer, The Hague University of Applied Sciences
• René Tieben, Principal, Cyber Defense at Capgemini Invent
• Mass Soldal Lund, Associate Professor, Norwegian Defence Cyber Academy,
Norwegian Defence University College
• Data Analyst, the Company
• Senior Security Consultant, the Company
• Principal Adviser, Advanced Technologies, the Company