Discussion and Conclusions

With the case studies complete, their results can be analysed. In this section, the knowledge questions set for validating the artifact are answered based on these results. Given the limited number of practitioners that participated in these case studies, no purely statistical analysis can be done. To assess the expected usefulness, usability and decision quality for decision- makers, the data gathered through the surveys in these case studies is used. Graphs summarising the outcomes for these indicators and their average scores have already been shown when discussing the individual case studies in Figure 34 and Figure 42. Refer to Appendix E for combined views of these indicators in both case studies. Histograms are also included to show the spread of answers given by participants. These give more insight beyond the mere average scores per indicator, such as insight in the spread of individual responses. These histograms have been normalised, meaning that they show relative frequencies. This denotes the proportion of responses that were assigned a certain value. The sum of the heights equal 1 for each data series. Note that the question scale used in the decision-quality related questions was different from the other indicators. This asked about agreement with statements rather than likeliness that a situation would be true in practice. The number of options and coding of scores was identical though. Nevertheless, an individual score regarding usefulness or usability can probably not be directly compared to a score for decision quality. The first and main question to be answered is about functional correctness: can the artifact be used to support decision-making in a microservice architecture? During the case studies, participants were indeed able to make decisions using the framework about several microservice-related challenges. These decisions were based on real-world scenarios, and only the number of variables involved were limited due to time constraints. Participants were not limited in the actual requirements and alternatives to be considered in their decision- making. Furthermore, no indications were found that the overview of challenges and their relations were incorrect. Participants understood the structure and steps involved and found these well-structured and compatible with their software development process. Given these outcomes, it is expected that the decision-making methodology of the artifact can indeed be used to support decision-making in a microservice architecture. A notable limitation to this generalisation is that not all challenges could be addressed in the two case studies. Even though the process of decision-making is the same for all these challenges, no empirical data is available yet to validate whether the findings in these case studies hold true for all challenges within the scope of the artifact.

In the case studies, the average scores given to questions about the perceived usefulness of the artifact were 1.00 and 0.96, respectively. This corresponds with the answer option of ‘somewhat likely’. In terms stated by Davis [82], this would mean that the participants thought

DECISION-MAKING IN A MICROSERVICE ARCHITECTURE PAGE 105/130

it is somewhat likely that using the artifact would enhance their job performance. The observations from the case studies seem to be in line with this score. Participants indicated that they thought the artifact could indeed support discussions about managing challenges in a microservice architecture and had favourable opinions about the pairwise decision-making and structured approach. On the other hand, decisions did in many cases take a considerable amount of time. This likely inhibited the perceived usefulness, thus contributing to the score on this subject not being higher.

The average scores regarding usability were somewhat higher; 1.29 and 1.50, respectively. Again, this most closely corresponds with the ‘somewhat likely’ answer option but is leaning a bit more towards ‘quite likely’ than the scores for usefulness. Therefore, looking at the definition given by Davis [82], it can be said that on average participants thought it is somewhat likely that using the artifact would be free from effort. The perception of usability was likely helped by the fact that participants recognised the clear steps in the decision-making process and the fact that the process is meant to be iterative as this closely resembled their day-to- day work activities. Participants also indicated that after some practice, they would likely be able to use the artifact in a real-world scenario. The biggest inhibitor to the usability score was tooling, as many participants indicated that more mature tools to guide the decision-making process would likely improve it.

The indicator that showed most difference between the two case studies is decision quality. The results of the first case study were somewhat disappointing in this regard, with an average score, yielding an average score of 0,00. As described, this was expected to be caused by unclarity about the selection and prioritisation of requirements during this case study. The artifact was changed to include more guidance on how to select the right architecturally significant requirements, and how to assign scores in pairwise comparisons. The expectation was that this would improve decision quality. The second case study – conducted with these clarifications in place – showed an improvement in the perceived decision quality with an average score of 1.21. Besides this, whereas in the results from case study 1 more than half of the responses given regarding decision-quality were negative or neutral, all scores given in case study 2 are neutral or positive. Furthermore, in the second case study there was also more consistency in the answers, showing a sample variance of 0.69 as compared to 2.74 in case study 1. This means that the average reported for case study 2 is more indicative of the dataset than that for case study 1, which showed much dispersion of answers. The more favourable score from case study 2 is no definitive proof that the additions to the artifact have improved the perceived decision quality by participants on their own. The score can be indicative of some improvement in this regard. It must be noted though that, even though the case study execution was kept as consistent as possible between the two case studies, the actual case used for decision-making did differ. This resulted in different decision being made, which may have also caused a difference in decision quality scores. Nevertheless, the observation that in case study 1 the requirements selection and prioritisation had to be redone and in case study 2 this was not needed, while in case study 2 the additions to the artifact aimed at improving these steps were in place, can be a sign of improvement. A fair conclusion to be drawn than is that on average, the participants were neutral to somewhat confident about the quality of the decision outcomes. Scores on perceived practicality were also low in the first case study, but better in the second one, with averages of -0.25 and 1.5, respectively. This difference can likely also be attributed to the uncertainty in the requirements selection and prioritisation in the first case study. During both case studies, no technical shortcomings to the framework were found. This also gives a favourable view of the usability in practice of the decision-making framework.

DECISION-MAKING IN A MICROSERVICE ARCHITECTURE PAGE 106/130

In section 8.1 the knowledge goals to be answered through the case studies were defined. These were as follows:

- Can the artifact be used to support decision-making in a microservice architecture? - To what extent do decision-makers find the artifact useful?

- To what extent do decision-makers find the artifact usable?

- How confident are decision-makers about the artifact’s decision outcomes?

- To what extent do decision-makers find the artifact and its outcomes usable in practice?

As discussed in this section, the observations and results from the two case studies give reason to believe that the artifact can indeed be used to support decision-making in a microservice architecture. Participants on average found the artifact somewhat useful, and somewhat to quite usable. They were on average neutral to somewhat confident about the decision outcomes and practicality, with indications to believe that the changes to the artifact to improve on this yielded more favourable results. No technical shortcomings that would inhibit the decision-making frameworks use in practice were found.

It is expected that these findings can be generalised to real-world use of the artifact, because of the case studies’ similarity to conditions of practice. The case studies were set up to be as similar to the real world as possible, and real scenarios were used as cases to be studied. Participants were merely limited in the number of variables involved, such as requirements and alternatives to be considered, not in their choice of these. As said before, little control was exerted over the treatment; the role of the researcher, materials and tooling was to educate participants about the decision-making framework and show its intended use, but not restrict them or intervene in their actual use during decision-making. In the case studies, not all identified microservice challenges could be considered. However, given that the process of decision-making is the same for all these challenges, and participants indicated no technical shortcomings to the overview of challenges, it is expected that the artifact is functionally correct for real-world scenarios.

The case studies and the answers to the knowledge questions presented in this chapter serve as the answer to RQ-5: How can the designed framework’s fitness for purpose best be validated?

Besides this, there were some qualitative insights that arose when running the case studies: - Allowing each participant to give their own rankings and then consolidating was seen

as helpful to reduce biases present in collective decision-making;

- Making pairwise comparisons using the AHP was received favourably and seen as supportive in making more informed decisions and evoking discussions;

- The outcome of the decision-making process can possibly be used to document and justify the decisions made – establishing design-rationale;

- The tooling used during the case study was seen as too complicated and could be more ‘hands-on’;

- The artifact was seen as practical and well-structured;

- The use case of using part of the framework when a system needs to be changed and certain decisions are already set was also foreseen by participants.

- Participants liked that they were ‘forced’ to think about the rationale behind their decisions.

DECISION-MAKING IN A MICROSERVICE ARCHITECTURE PAGE 107/130

These insights, together with the outcomes of the surveys and the observations done during the case studies can be used to further improve upon the artifact in the future. Most often, the need for better tooling was mentioned as inhibiting the decision-making framework’s potential to be shown. Nevertheless, participants were convinced that many parts of the framework would be useful for helping them manage challenges when developing microservice architectures.

DECISION-MAKING IN A MICROSERVICE ARCHITECTURE PAGE 108/130

9

DISCUSSION