Relation to Evidence-based Computing

When looking at Figures 3.4 and 3.5, it becomes evident that the Search Interaction Op-timization methodology is a concretization of an even more underlying principle called

3Cf. http://scrummethodology.com/ (Feb. 26, 2015).

34 Chapter 3 Proposed Solution: Search Interaction Optimization

Evidence

System

Evaluate Optimize

Fig. 3.5.: Rough sketch of the principle of evidence-based computing based on Gaedke (2013).

evidence-based computing. The latter has been introduced by Gaedke (2013) and follows an abstract, iterative process:

— A given system (e.g., a website or desktop application) is evaluated, whereas the kind of evaluation practice is up to the developer (Gaedke, 2013).

— From this evaluation results an artifact in terms of well-founded evidence (Gaedke, 2013). Such evidence could be, e.g., a scientifically substantiated report, a set of metrics, or a filled in checklist.

— Any optimization applied to the system in the next step is directly based on the collected evidence (Gaedke, 2013). Mathematically speaking, optimizations applied to the system in this step of the process are a function of the evidence as well as the previous state of the system: system^′= optimize(evidence, system).

— More evidence is collected in the next iteration, to either validate the applied adjust-ments or further optimize the new state of the system (Gaedke, 2013).

— The iterative process is repeated until a satisfactory state of the system is reached (Gaedke, 2013).

The main advantage gained through the application of evidence-based computing is the fact that optimizations or adjustments to an existing system are highly substantiated rather than based on educated guesses or speculation (Gaedke, 2013). Hence, the principle serves as an adequate basis for our Search Interaction Optimization methodology, as it—among other things—by design facilitates objective communication with company officials (SIO3 ✔). Yet, evidence-based computing is a rather generic concept that also must not necessarily fulfill several of the core principles of our methodology stated above. To give just one example, the concept assumes that at least an initial prototype is present before entering the iterative cycle depicted in Figure 3.5 (Gaedke, 2013). That is, no particular support for creating a new system from scratch is provided (SIO7 ✘). Therefore, evidence-based computing is a well-suited starting point but must be seen as a generalization of our novel methodology.

The precise is a–relationship between these two is shown in Figure 3.6.

3.4 Solution Concept 35

EvidenceBasedComputing

{abstract}

evaluate() optimize()

SIOMethodology

{abstract}

evaluate() optimize() create()

SIOToolkit

evaluate() optimize() create()

<<interface>>

EvaluationComponent

evaluate()

<<interface>>

CreationComponent

create()

<<interface>>

OptimizationComponent

optimize()

0..*

1..*

1..*

1..*

Fig. 3.6.: UML class diagram visualizing the relationships between evidence-based comput-ing and the Search Interaction Optimization methodology and toolkit. Compo-nents of the toolkit can also be used standalone; thus the0..* quantifier.

36 Chapter 3 Proposed Solution: Search Interaction Optimization

3.5 Summary

SERPs whose designs have been driven by mainly company-centric decisions demand for a new approach ensuring more usable web interfaces in this respect. Therefore, we have introduced the Search Interaction Optimization toolkit, which aims at addressing the prob-lems identified in the previous chapter. It shall provide adequate means to developers for developing more usable SERP interfaces that ultimately lead to satisfied and loyal searchers.

From the proposed toolkit, we have moreover derived an eponymous methodology that extends the principle of evidence-based computing (Gaedke, 2013). The novel methodology defines seven requirements that must be fulfilled by any corresponding implementation, thus particularly by the toolkit designed and developed in the remainder of this thesis. In the following we will investigate the primary hypotheses introduced above and describe the development of the Search Interaction Optimization toolkit according to Figure 3.3.

In particular, each chapter describing one of the necessary components will elaborate on existing approaches and highlight the novelty of our approach. Moreover, we will assess the status quo in the industry context of this thesis to further strengthen the requirements already posed. In the next chapter, we start by introducing INUIT—a new minimal instrument for determining the usability of web interfaces.

3.5 Summary 37

4

I NUIT : The Interface Usability Instrument

After having identified the need for better evaluation and optimization of SERP interfaces and introduced the concept of the SIO toolkit, we are now starting with the description of the design and development of the necessary components. First off, Rey—our developer persona—requires means for correlating the implicit user feedback in terms of interactions she is collecting on SERP interfaces with aspects of usability.

Hence, the following chapter¹presentsINUIT—a novel usability instrument that has been specifically designed for meaningful correlation of its contained items with client-side interactions. INUIT serves as a basis for more elaborate components—i.e., WaPPU and S.O.S.—but is also a stand-alone tool that can be applied in a variety of usability evaluation contexts. The instrument intends to fulfill Requirement 2.1 and to cover the complete evaluation of usability as defined in Section 3.3.

Front End Back EndINUIT 2 3 4

Fig. 4.1.: Progress at current point in thesis (Chapter 4).

4.1 Introduction

One of the core aims of this thesis is to provide a method for web interface evaluation that can compete with the efficiency of split testing while being more effective in measuring usability. A straightforward approach would be to make use of real users’ interactions with a web interface to infer knowledge about its usability. Optimally, such knowledge would be present in terms of a key performance indicator (i.e., a usability score) for easier communication with stakeholders who are not usability experts.

To be able to realize such a framework (Figure 4.2), it is necessary to build upon an adequate usability instrument for providing a quantitative measure that combines ratings of the contained items. For instance, usability = −(confusion + distraction). As usability is a latent variable, we need to define factors thereof that can be meaningfully inferred from interactions, e.g., faster and more unstructured cursor movements indicate user confusion ⇒

1Earlier versions of parts of this chapter have been published as Speicher et al. (2013b) and Speicher et al. (2015b).

39

Live

Fig. 4.2.: A model providing a quantitative metric of usability.

confusion = 1. Numerous instruments for determining usability have been developed (e.g., Brooke, 1996; Fisher et al., 2004; Green and Pearson, 2006; Palmer, 2002), but none has been specifically designed for providing a key performance indicator for usability that can be directly inferred from user interactions.

Thus, we propose INUIT—a new usability instrument for web interfaces consisting of only seven items that have the right level of abstraction to directly reflect users’ client-side interac-tions. In the following, we describe the current state of the art as well as our assessment of usability evaluation practices in the industry context this thesis was embedded into. Subse-quently, we derive three requirements a usability instrument has to fulfill to be adequate for our research aims. After that, we provide insight into the two-step process of determining the items of INUIT. First, we have reviewed more than 250 usability rules from which we created a structure of usability based on ISO 9241-11 (ISO, 1998). Second, we conducted semi-structured expert interviews with nine experts working in the e-commerce industry.

Based on a user study with 81 participants, results of a confirmatory factor analysis show that INUIT’s underlying model is a good approximation of real-world perceptions of usability.