Framework for Selecting U-I Data Collecting Techniques

Table 4.2: Evaluation criteria for U-I data collecting techniques

Criteria Description

Timeliness Time-wise considerations on the availability of the data the technique produces.

Targets Technique’s support for collecting multiple types of data and/or for varying purposes and stakeholders.

Scalability Describes the work effort required from the developers in implementing additional collecting places to the source code.

Overhead Effects of the technique to the performance in the field and possible downtime during implementation.

Sources Flexibility of collecting data from different sources, such as various platforms, applications, or from applications of different languages. Configurability Considerations on how configurable the technique is, e.g. switching

the collecting on and off and between different types of data. Security Trustworthiness of the organization and of the solution behind the

technique.

Reuse Considers if the technique provides a one-time solution or if it can be reused easily with another application.

Change Describes the levels of change required to the architecture and/or environment of the application.

The framework for selecting U-I data collecting techniques consists of two parts. Firstly, it includes a set of criteria for evaluating the different U-I data collecting techniques. Secondly, we have designed, refined and validated a method that describes the steps of selecting a suitable U-I data collecting technique.

The first published evaluation criteria set was presented in Publication II. At that point, the set was based on a literature review and a case study with one company. To make it more actionable for practitioners, we then used the set with three software teams in the actual choosing of a U-I data collecting technique. Based on these three cases, we then refined the criteria set where necessary as described in Publication III. Consequently, we

4.1. U-I Data Collecting Techniques 31

claim the following set of criteria, presented in Table 4.2, validated within the case study teams.

Define

a main goal

Remove

irrelevant

evaluation

criteria

Find out

critical

limitations

Reject

unsuitable

techniques

Prioritize

remaining

evaluation

criteria

Evaluate

remaining

techniques

Figure 4.1: Method for selecting U-I data collecting technique. Adapted from Publica- tion III.

The second part of the framework, i.e., the method for selecting U-I data collecting techniques, is summarized in Figure 4.1. We developed it similarly to the set of evaluation criteria in the same Publications II and III. The method consists of six steps, which describe the activities we went through with the case teams in the processes of selecting a suitable U-I data collecting technique. The steps are as follows:

1. Define a main goal. A clear vision of why the collecting technique needs to be implemented is a critical part of the selecting method. Defining a goal based on such vision is thus arranged as the first step in the method. It guides the decision making in the rest of the steps. For example, a goal can be defined as "Try out A/B

testing as a one-time experiment in one application".

2. Remove irrelevant evaluation criteria. Not all of the evaluation criteria are required in every case. To make the focusing on the important ones somewhat easier, the irrelevant can be removed in this step. At the same time, the step guides one to familiarize themselves with the evaluation criteria. Continuing with the example from step 1, evaluation criterion such as "reuse" could be removed. It would make no sense to invest in a reusable implementation of a technique if it is intended for a one-time solution.

3. Find out critical limitations. Before more detailed evaluations, the critical limitations to the collecting techniques by the application, environment, and or- ganization need to be examined. These include case characteristics, such as the used programming languages, privacy regulations, and production environment technologies. For example, a regulation of not storing user related data outside a specific geographical region can set a critical limitation to the selecting of collecting technique.

32 Chapter 4. Results

4. Reject unsuitable techniques. At the latest at this step, one should familiarize themselves with the available U-I data collecting techniques. The step is highly based on step 3. It uses the emerged limitations in analyzing which of the techniques can be removed at this point. Continuing the example of step 3, a collecting tool developed by a third party might need to be rejected if it remains uncertain if that party always keeps the data within the suitable region.

5. Prioritize remaining evaluation criteria. This step is highly intertwined with step 1. The defined main goal for the U-I data collecting sets the stage for this step and assists in deciding how the remaining evaluation criteria should be prioritized. Again referring to step 1 and the example of a one-time collecting experiment, a software team might prioritize a low level of change required to their software architecture over a low scalability of the technique.

6. Evaluate remaining techniques. At this final point, the remaining techniques are evaluated in terms of the prioritized set of criteria. The result of taking all the steps should be an understanding of how the collecting techniques suit the case at hand.

The presented framework is not dependent on specific techniques for U-I data collecting. Consequently, it can be used also with techniques that have not been yet studied by us. However, we have identified five collecting techniques both to assist practitioners in a more concrete manner and to test the framework in a practical context. In the following subsection, we present these identified techniques with the help of the framework.