Learning from Sketching

Our usability tests clearly showed that a wizard needs to be able to follow and control the dialogue flow. In trying to tackle this problem, we were not only interested in a solution that would help the one scenarios we built our initial prototype for, but we were also interested in searching for a generic interface concept that could be applied in a range of different settings (cf. Table 4.2). To do so we used paper prototypes and sketches as a means of quickly designing and comparing ideas.

Paper prototyping and sketching are easy and cheap methods to gain early feedback on design. They help to develop ideas and discuss different possible solutions. Through sketches,

designers act like architects, when they find new relations and features whilst viewing their own sketch (Suwa and Tversky [1996]). Sch¨on [1984] argues that the designer reflects-in-action. Thereby it is not just the actual drawing act, but the way in which the situation talks back to the designer and lets her change initial thoughts. This intrinsic process of drawing and at the same time communicating with the sketch, defines the designer’s understanding of the situation. In our case we used sketches to compare different layouts of possible generic wizard interfaces. We created paper-based interfaces for each of the four scenarios defined earlier (cf. Table 4.2). Going through this process we were able to identify three main layout concepts that seemed to be applicable. These concepts are discussed in the following sections.

Dialogue

All of the layouts we designed had in common that the main area of the screen was dedicated to the running dialogue. In our understanding, the wizard’s task mainly consists of choosing from a predefined set of dialogue utterances. Supporting this selection process would significantly simplify the role of the wizard. In order to do so our goal was to decrease the number of possible utterances to choose from. Comparing different designs we found that subdividing the dialogue into smaller steps seems a valid solution for this problem; one that could be applied to all the scenarios we designed for. Using a tab structure we were able to split the dialogue into different stages but at the same time we maintained the overall structure, which may help the wizard to keep track of the dialogue progress and also allows for quickly switching between different dialogue stages (cf. Figure 4.3).

Figure 4.3– A tab structure representing the dialogue and its stages may help the wizard to choose from a predefined set of response utterances.

Recovery

The second important concept we identified as being relevant for an intuitive to use wizard interface, is the support of the common problem of error recovery. Since there is a probability of misunderstanding in any kind of dialogue situation, a wizard needs to be offered a way of recovery. Recovery might be gained through re-sending the last utterance or by using prede- fined repair utterances that would invoke a test participant to repeat or specify her last statement (e.g.“Sorry, I did not understand you. Could you please repeat.”). The problem of predefined repair utterances is, however, that they do not occur at a certain point in time. Rather a wizard needs to be able to use them spontaneously within the running dialogue. As a generic solution for this, we propose a separate area for repair utterances that would be constantly visible to the wizard independent of the dialogue stage (cf. Figure 4.4).

Figure 4.4– A seperate area for repair utterances should permit the wizard to act spontaneously.

History

Finally, the third aspect we found to be helpful for showing the progress of the dialogue flow is to introduce a history function. The idea here is that a wizard might want to see what utterances she has already used and possibly what a test participant has replied. Also, incoming and outgoing utterances should be easily distinguishable. That is, a wizard should be able to see what came from a participant and what she herself, acting as a wizard, has sent. As one way of tackling this problem we propose the use of symbols and different colours to differentiate between wizard’s and test participant’s utterances (see Figure 4.5).

Figure 4.5– A history using different symbols and colours to differentiate between wizard and subject statements might support the wizard’s task.

4.4

Summary

Having completed this initial phase of building a prototype and evaluating its usability, and having reflected upon the results of these tests with the described sketching exercise, leading to concrete concepts for a generic wizard interface, we conclude this primary stage of our research program. Summing up, we started our exploration route with an extensive discussion of existing tools and what we can learn from them. Then we moved on to identifying global recommendations for running WOZ experiments, and finally we created an initial prototype of a WOZ tool supporting a single test scenario. The tool was evaluated using a small-scale usability study and results were reflected by a set of sketches aiming for the design of a better wizard interface.

All this serves as the basis for the next stage in our research agenda which looks more closely at generic WOZ tool support. To do so, the next chapter of this thesis analyses the WOZ method from two different perspectives. First, we re-examine the literature and look more closely at the user (i.e. wizard) side of the experimentation. As part of this, we report on the results of an interview study that aimed at identifying some of the wizard challenges that are not usually found in publications. From there we move on to the system perspective of WOZ, and look more closely at technological aspects of tool support. The goal of this whole process is to move away from the here described prototypical WOZ tool for a single scenario, to a generic platform for WOZ experimentation that is employable for a variety of different experiments.

Generic Support for Wizard of Oz

Experimentation

“Get an oil-can and oil my joints. They are rusted so badly that I cannot move them at all; (...)” – The Tin Man

This chapter analyses generic support for WOZ. Doing so, we look at two different per- spectives of WOZ experimentation. We start at the user perspective that analyses people’s interests in employing the method (cf. 5.1), and then move on to the system perspective which aims at defining wizard and technology dependent requirements (cf. 5.2). The insights gained from this analysis forms the necessary foundation for offering generic tool support.

5.1

User Requirements

By looking at the user perspective we are interested in the distinct goals and interests re- searchers follow when running WOZ experiments. Reflecting on what we had seen in the literature and other reported examples, we start with a categorization of potential users and their motives for employing the method. In a next step, we expand on this theoretical analy- sis and discuss the results of an interview study in which we contacted researchers from both academia and industry and asked them about their experience with WOZ.