the first artifact, the design for the unobtrusive analytics technology stack presented in Publication I. The demonstration and evaluation phases of the DSRM process were conducted with a case study, where we used the analytics stack with Vaadin framework2 and its demo application3. This work was done in the early 2015. The objective of the first iteration was to produce a showcase artifact. With that we could demonstrate the collecting of U-I data and its utility for interested organizations in the N4S program. After a successful demonstration with the first artifact, we got to conduct a single case study with organization A in the fall of 2015. There, we defined a new objective for the second DSRM iteration. The selecting of an appropriate U-I data collecting technique was not an easy task. Therefore, we decided to assist this by developing the second artifact, i.e., the selection framework. This was presented in Publication II. Then, we were able to conduct a multiple case study with organizations A, B, and C from February to December 2016. The objective of this third iteration remained the same as in the second iteration. However, we were able to focus on the evaluation of the artifact. This resulted in a few adjustments and refinements in the selection framework and these are presented in Publication III.
3.1.2
Action Design Research
Similar to DS, ADR seeks to generate prescriptive design knowledge by developing and evaluating IT artifacts [18]. However, ADR adds an organizational setting to the research. It mixes the evaluation of the artifacts tighter into the method, whereas DS had the evaluation as its own separate phase. Like DS, ADR addresses the class of problems rather than solving immediate case specific problems [18]. The main difference between DS and ADR is in the researcher intervention. The ones without researcher intervention in organization could be looked as DS. With such studies, the intervention can still happen in the evaluation phase. The studies with an intervention, on the other hand, appear more like ADR.
We used ADR for guiding our work in the second research segment. The study is presented in Publication V. The built and evaluated IT artifact of that study was the U-I Data Utilization Method. Although the case organizations were the same as in the previously presented DS guided multiple case study, we intervened with the work of the case teams more clearly. The teams had either no or very little previous experience with using U-I data for software development, and our study with them started such efforts. The U-I data utilization method models the work done in and with the case teams during the research period of February 2016 to February 2017.
3.2
Research Methods
To conduct the research guided by the chosen research strategies and to add to the knowledge base of software engineering research, we mainly used the case study method. In addition, we conducted one study with the survey method and one publication (Publication VI) was a concept paper without a research method. In the following, we will present these methods that we used for conducting the research of this doctoral thesis.
2https://github.com/vaadin/framework 3https://github.com/vaadin/dashboard-demo
24 Chapter 3. Research Design
3.2.1
Case Studies
Yin [111] describe case studies as rich and high detailed descriptions of specific instances of a phenomenon, and that they are usually based on multiple data sources. Runeson and Höst [112] distinguish between four types of case studies by their purposes. Exploratory studies are aimed at finding out what is happening to generate knowledge and to produce new ideas and hypothesis for future research. Descriptive studies focus on portraying a situation or a phenomenon. Explanatory studies seek explanations to a situation or problem. They are mostly but not necessarily of the form of causal relationships. Confirmatory case studies are also seen as explanatory. Finally, some case studies are intended to improve a certain aspect of the studied topic [112].
During the doctoral thesis research, we used many of the different types of case studies. All of the studies in the first research segment were case studies, but all of them were of different type. The first one could be defined as an improvement study since it investigated the characteristics of a new U-I data collecting technique. The second was an exploratory case where we examined a specific situation in one organization. We used its results as a basis for a new development phase for the research artifact. The third case study was then an explanatory study that was aimed at evaluating and confirming that artifact. Finally, after a year from the ADR research period that led to the development of the U-I data utilization method we studied the same case teams on how they had actually utilized U-I data on their own. This could be looked at as a descriptive case study on its own, although it was published as a part of the whole ADR study in Publication V.
3.2.2
Data Gathering and Analysis Methods
We used a wide scale of research data gathering methods during the thesis work. Time- wise, the most used method was to organize, participate in, and/or examine the memos of different workshop meetings. Three types of workshops were held during the studies either with or by each case team. Firstly, the researchers motivated the teams to consider U-I data collecting as a new possibility to them and discussed the options for collecting techniques. Secondly, two of the teams had internal brainstorming workshops for coming up with objectives for U-I data. One team had such a session with the first author of this thesis. Thirdly, after collecting the U-I data in each case, either a presentation of its results or a workshop for analyzing the results was held together with the team members and the first author. All of the above workshops were of unstructured nature. In addition to the research data gathered in these workshops, the first author of this thesis had designated work desks in the same rooms where the software teams were working in cases A and B. This allowed us to explore the contexts of the study thoroughly and to exchange information also informally with the team members. For example, this setting allowed us to give concept presentations within the organizations.
Additionally, we used a questionnaire survey for the study presented in Publication IV. The questionnaire was sent via email to four Finnish software consulting companies small to medium in size. The questionnaire included five open-ended questions about PDD and a few about the background of the respondents. 25 responses were given. We also interviewed the case teams’ practitioners formally for the final part of the ADR study presented in Publication V. To allow a broad range of answers, the interviews were semi-structured and we mainly used open-ended questions. Altogether we had three interviews, one for each case. Two researchers were present in all of them and the number of interviewees was three in case A and one in cases B and C. The interviews were recorded