On a practical dimension this research was seeking to obtain the relevant knowledge to cre- ate a prototyping tool that can be employed beyond the frontiers of a single experiment. We envisioned a tool that tackles the WOZ method on a more generic level and therefore offers high flexibility and ways to customise. While usually a specific research interest triggers the development of such a tool, our motives for this venture were different. We were not driven by a single research question that might require the conduct of a specific WOZ experiment to be answered, but we analysed a variety of different WOZ settings to provide us with the relevant requirements that would need to be supported by a generic tool implementation. Our research therefore relied on third party research interests which gave us the ability to study people’s motivations for using WOZ and helped to identify the challenges they were facing by doing so. Previously published work was therefore our starting point for building a robust conceptual framework.
Reviewing existing tools helped to gather a basic set of requirements and provided a broader view on different use case scenarios to be supported. Reviewing the literature also provided a general impression of the type of problems a wizard is facing when running experiments. Previous experiments and the tools that were used to run them gave us an overview of what researchers/designers experienced when confronted with the wizard task, and what tool func- tionalities were integrated to help them.
However, the literature was only the starting point which led to the definition of an initial set of requirements for the tool. Additional information came from both acting as a wizard (learning by doing) as well as observing other wizards when running real experiments. Since the goal was to create a tool that, while generic enough to be used in a variety of settings, could be customised to support the very specific demands of single wizards, we were highly dependent on feedback provided by those real users employing different versions of our tool in real experiments. Methods from UCD were employed to collect this sort of interaction data. While formal usability tests (cf. Dumas and Redish [1999]) were used to identify severe problems and conceptual misfits with respect to the wizard interface, it was the observation of
longer term interaction with the tool that showed missing functionalities and highlighted those challenges that were difficult to master even with sufficient hands-on experience.
Two different types of evaluations were conducted. Firstly, we employedwithin-experiment
analysis, where the feedback was gathered after different experiment runs that belonged to the same experiment setting and whose results were used to improve upon existing tool function- alities. Secondly, we used the differences between experiments (between-experimentanalysis) to identify new requirements leading to the expansion of tool functionalities. Furthermore we obtained feedback from different perspectives. That is, while at the beginning our own impres- sions collected from acting as a wizard guided the development process, later on we expanded the tool and integrated functionalities based on observations and discussions with other, third party researchers performing this role.
This iterative development process was employed to move from an initially hard-coded prototype that was able to run a single defined experiment to a platform that supports various forms of WOZ experimentation. Development and evaluation phases of this process were highly integrated so that new functionalities were driven by direct user demands rather then top-down engineering decisions. We may therefore summarise that, triangulating between what was presented in the literature (tools and scenarios), what we experienced ourselves when acting as a wizard, and what we learned from other third party wizards drove the design and development process for this generic WOZ prototyping tool.
3.4.1 Learning from Existing Tools
Before any actual prototyping and development work for the envisioned WOZ tool could start we looked at existing solutions and what features they offer. An essential part of this analysis was to not only enumerate these functions but also to understand why they had been imple- mented. As most (if not all) work on WOZ tools is grounded in scientifically motivated projects the associated research papers were an obvious starting point for building up this understand- ing. As already pointed out earlier, most of these applications were built for very specific experimental conditions. Those experimental conditions can be seen as a basis for identifying distinct user requirements. We wanted to understand the reasons for the different experimental set-ups in order to be able to map them to distinct features integrated by the employed tools. While existing tools mainly focus on their specifically associated experiment conditions, our goal was to offer more generic prototyping support. Hence we had to identify and understand a variety of tool requirements which required a comprehensive analysis of applications previ- ously presented. We specifically wanted to learn from the literature what types of functions essentially define a WOZ tool, how they are integrated and what role the wizard plays when using them. Furthermore it was our goal to clearly identify those features that were imple- mented to directly support the task of the wizard. By understanding those features integrated in existing tools and what motivated them, we were able to define the baseline of the requirements that needed to be supported.
3.4.2 Learning from Previous Experiments
Despite our overall goal of achieving generic support it seemed important to start with specific application scenarios that could be taken as a road map for further expansion. We used per- sonas and scenarios, as described in the UCD process, to find valid use cases for WOZ. These scenarios served as an important information source for successfully running experiments and also helped define the boundaries for our first round of prototype development. Again, exam- ples reported in the literature provided a valid starting point while additional research interests expressed within our research group highlighted additional potential use cases. The overall goal of this step was to define a list of guidelines for WOZ experimentation and to identify a set of typical application areas and concrete scenarios where the method can realistically be employed.
Doing so was not only helpful to better understand the application landscape our tool would need to cover, but also provided us with distinct requirements that were used to drive the design process for the wizard interface. As can be seen from the multitude of tools presented in the past, tools for different scenarios offer different functionalities. Furthermore they show distinct differences when it comes to the layout of their wizard interface. Our goal was to find commonalities. We wanted to define a generic interface that would support wizards in a range of experimental settings. As such we were aiming at general layout concepts that would suit several application contexts and would also allow for the necessary flexibility to adapt to the different stages of WOZ prototyping. Looking at examples presented in the literature and expanding upon them helped to explore the existing design space and provided a pathway, which was consequently used to reveal system requirements that go beyond a possible baseline definition.
3.4.3 Learning from Building Prototypes
Based on what we had learned from existing applications and taking their distinct scenarios as a guideline we were able to start the development of the tool. In accordance with SDM and UCD, we began with paper sketches and other simple prototyping techniques in order to explore initial design ideas. While these early stage concepts were primarily used to compare and reflect upon the different solutions of a possible wizard interface, they also served as an im- portant basis for defining the underlying tool architecture. As a flexible architecture is key for achieving generic support, this aspect of tool design was essential. That is, if the technological base would not allow for easy expansion and customisation it would also be difficult to offer the relevant functionality on an interface level. Existing tools highlighted some of the chal- lenges involved when one aims for flexibility and so constantly re-visiting the literature helped to learn from them. This also shows the iterative nature of this design methodology which requires a continuous dialogue between the tool perspective and the application perspective of software development. While the tool perspective is often found in demonstrations and on- line documentations, the application perspective is represented by the variety of scenarios that
have been using the WOZ method. They provided the necessary use cases that our tool had to support. Eventually elaborating on one of them, we started building a first working prototype which served as a technology probe.
Starting a first round of evaluation, the technology probe was tested in a realistic setting (i.e. it was employed in a WOZ experiment that aimed at answering a distinct research ques- tion) leading to first insights on how the designed interface is operated in real time. Here the realistic scenario was a crucial aspect of this evaluation process as it created the necessary con- text, which made it possible to obtain valid feedback. Furthermore, as WOZ experimentation usually requires a number of test runs, it also made it possible to see whether a user was able to adapt to new interaction paradigms or if they would keep causing problems over time. A number of iterations involving new prototypes, modifications and extensions as well as varying scenarios and accompanying users tests, were necessary to move towards our goal of generic WOZ support. In doing so, evaluations consisted of both studies of the tool used in differ- ent WOZ experiments as well as formal usability tests conducted with participants who were not following a dedicated research agenda. While the former provided the relevant data for improving and expanding upon the functionality of the tool, the latter was seen as a valuable indicator for intuitiveness of use; an important aspect since one of our goals was to make WOZ prototyping more accessible to non-expert users.
In summary, in order to explore WOZ tool support, the first part of this thesis effectively followed an integrated SDM-UCD approach, which grounded its design decisions in informa- tion gathered from the literature as well as in the observation of people (including ourselves) using different prototype versions of a potential tool instantiation. The next section will elab- orate on that by describing the second, rather theoretical dimension of this integrated research agenda, which more thoroughly focused on generating new insight into WOZ experimentation in general, and the challenges of the wizard task in particular.