Validation Methodology

As Wieringa states, “To validate a treatment is to justify that it would contribute to stakeholder goals when implemented in the problem context” [4]. For this, a model of the artefact interacts with a model of the intended context that both resemble their real-world counterparts. This is illustrated in Figure 26. The interaction between the model of the artifact and the model of the context is then observed to build a theory about how this interaction would work in real-world scenarios. This theory can then be used to make generalisations and predictions about the observed phenomena.

Figure 26 - Relation between Validation Model and Target - Adopted from [4]

The validation will be done through two case studies in the form of single-case mechanism experiments. For the case studies at Thales Naval, several software architects were asked about what current projects they and their team were currently working on that involved decision-making in microservice architectures, and which of those they would see fit to serve as model case for this research. The current description of the artefact acts as model of the future implemented artifact and is applied to two cases that resemble practical scenarios that will be used as models of the intended context. The guidance given in chapter 18 in Wieringa’s book [4] on how to structure such experiments will be used to design the case studies. Those parts that both case studies share are discussed together, and case-study specific aspects – mainly the model of the context – will be explained in their respective sections.

The knowledge goal in these case studies is validation of the designed artifact or treatment. The higher-level goal of the treatment is to improve decision-makers’ work in managing design challenges related to communication between, integration and management of microservices. The knowledge context consists of the problem overview, background literature and treatment design discussed in present research.

The conceptual framework of this validation research consists of the problem description, theoretical background and artifact design as well as the measures for operationalising indicators described in this research. The generalisation is that the steps in the decision-

DECISION-MAKING IN A MICROSERVICE ARCHITECTURE PAGE 89/130

making framework contribute towards decision-makers’ goal of better managing microservice design challenges. The generalization predicts that this will happen when the decision-making framework is used. Knowledge questions that need to be answered in these case studies are about whether the designed artifact satisfies the goal-level requirements described in section 6.1.2. Besides this, there is the question of whether decisions on the identified microservice challenges can actually be made using this decision-making framework; i.e. the functional correctness needs to be verified. The designed framework’s usefulness, ease of use and the quality of the decision outcome are the main factors in contributing to the stakeholders’ goal of better managing design challenges related to communication between, integration and management of microservices. These can be expressed as follows:

- Can the artifact be used to support decision-making in a microservice architecture? - To what extent do decision-makers find the artifact useful?

- To what extent do decision-makers find the artifact usable?

- How confident are decision-makers about the artifact’s decision outcomes?

- To what extent do decision-makers find the artifact and its outcomes usable in practice?

The intended population is the set of microservice architecture design projects that:

- Are executed in an organisation with an iterative or agile-like software development process;

- Have a clear business strategy and project goals set at the start of the project; - Employ a group facilitator to lead the decision-making process.

The object of study is the first design of the artifact and will be applied in two different but fairly similar case studies, the specific of which will be described in their respective sections. These cases do not support directly gathering data for mathematical analysis. Questionnaires to be filled out by participants can be used to get an overview of the outcomes of using the artifact. Furthermore, qualitative data such as the decisions itself, as well as remarks by the participants during the process can be used to describe and possibly explain some of the outcomes. The model will abstract from the real world in a number of ways. First, the number of participants, requirements, decisions and solutions to be considered will be limited because of time constraints. Ideally, one would ask an entire team to join and make decisions for an entire microservice architecture in as much time as needed. However, such a commitment of organisation resources cannot easily be expected for this experiment. Furthermore, the case studies will be conducted at Thales Naval. Even though the organisation is part of the intended population, it is not necessarily the de-facto model for the population. Nevertheless, it is expected that many of the observations done in the Thales Naval context can be generalised. Both case studies reside in a context that resembles a real-world scenario. The decisions to be made are similar to those that decision-makers face more often, but this time the designed decision-making framework will be used to structure this process. The process will be guided by the researcher, who will act as facilitator. An introduction to the decision-making framework will be given in a presentation. The actual decision-making is supported with the tooling described in section 7.5. Little control was exerted over the treatment; the role of the researcher, materials and tooling will be to educate participants about the decision-making framework and show its intended use, but not restrict them or intervene in their actual use during decision-making. This should best support analogic inference to the real world.

DECISION-MAKING IN A MICROSERVICE ARCHITECTURE PAGE 90/130

The case studies will start with an introduction by the researcher, explaining the decision - making framework and its intended use as well as the concept of pairwise comparisons through a presentation. The goal of the artifact to mainly be of a supportive role in the participants’ day-to-day work will also be highlighted. This is done to clarify that it would not tell them how to do their job but aims to make it easier. The case to be studied will be explained to ensure that all participants would be on the same page about the system to be discussed. A check will also be done to assess whether all participants are familiar with microservices. Finally, the participants are informed that because of the limited time available for these case studies, the number of requirements, decisions and alternatives to be considered would be limited. Nevertheless, they will be encouraged to approach the process in a way that they also would in the real world, to make it as true-to-life as possible.

In both case studies, data on usefulness, ease of use and decision-quality will be gathered by letting participants fill out questionnaires. The questions defined by Davis [82] in their paper describing the Technology Acceptance Model (TAM) can be used for measuring the first two variables. The question of confidence comes down to measuring decision quality. No detailed information about previously made decisions in microservice architectures and their performance is available. It is therefore hard to objectively judge decision quality based on the performance of decision outcomes in practice. As an alternative, this can be done using the six elements of decision quality described by Spetzler [83]. These questions have been augmented with three additional ones, directly asking participants about how confident they feel about decisions, the practicality of the decisions and about the contribution of the artifact to making these decisions. The question about the perceived practicality, together with asking about the framework’s technical correctness are used to assess practicality. An overview of the questions used for these measurements can be found in Appendix D. From this data, descriptive statistics can be derived to gain insights in the participants’ views on the decision- making framework. The data from the questionnaires will be mainly used to support descriptive inference. Furthermore, the events, remarks and outcomes of the decision-making process can be used to further analyse the model artifact’s effects in the model context. An attempt will then be made to generalise from this.