The surveyed research indicates a strong prevalence in the research for the web and user generated data domain with software engineering focused on in only
one paper. Tag cloud visualisation itself has not been as extensively evaluated as other areas, indicating there is still room to define their overall effectiveness and develop ways to improve the tag cloud as a technique. A large proportion of papers evaluated interactive interfaces tailored to particular datasets, popula-tions or mediums. We should note that no interface identified in the mapping study proposed a system such as Taggle, where data fields from a multi-variate dataset are mapped to tag cloud visual properties and manipulated interactively.
Moreover, despite the prevalence of interactive interfaces, evaluation approaches were of a limited range — predominantly visualisation use techniques measuring user responses times, as opposed to strategies that consider the data analysis process, which are of high relevance and value when evaluating tools with data exploration and knowledge discovery goals.
In recent years there has been a spate of research surrounding tag cloud visu-alisation. This systematic study of 60 papers (from 2007 to 2012) was undertaken in order to discover what sorts of topics relating to tag cloud visualisation have been evaluated and to what extent. This work provided an overview of what is known about tag clouds, and helped us plan and focus our overall evaluation strategy which is presented in Chapter 6.
Evaluation Strategy for Taggle 6
Our systematic mapping study of tag cloud research (presented in Chapter 5) identified tag cloud visualisation as a technique had not been as extensively eval-uated as other topics (such as interactive interfaces incorporating tag clouds), in-dicating opportunities for measuring effectiveness and developing ways to improve the tag cloud approach. No research identified in the mapping study described a system with interactive features like Taggle, where data fields from a multi-variate dataset are mapped to tag cloud visual properties. There were a limited range of evaluation approaches despite the widespread presence of interactive interfaces, typically utilising user performance measurements (visualisation use type tech-niques), rather than strategies considering the data analysis process, which are of high practical value and relevance when evaluating systems to explore data and discover information. We were therefore encouraged in several aspects: by focusing on software engineering we were promoting use of tag cloud visualisa-tion outside the typical web and user generated data domains; that our tool was unique in its approach, and that to conduct an evaluation with broad-ranging
goals we should consider multiple targeted evaluation strategies.
Based on the results of the systematic study, this chapter outlines the process and outcomes of designing an evaluation plan for Taggle. We describe mapping out our overall evaluation strategy and associated methodologies in §6.1, and evaluations that were selected to be conducted in §6.2. Generation of suitable datasets for experimentation is discussed in §6.3. In§6.4 we describe the process we took to conduct three of our experiments on an eye-tracking machine.
6.1 Overall evaluation strategy
Our overall goal for evaluation was to explore the potentials and limitations of our tag cloud visualisation tool Taggle. We decided to map out sample areas of the tool which were of interest, in order to strategically plan a broad-ranging series of evaluations. This map is represented in Figure 6.1 on the next page.
Potential areas of evaluation were divided into the part of the system being looked at — visual encoding or interactive tool. Each potential research question was associated with an appropriate evaluation strategy (highlighted in blue): User Performance, User Experience, Visual Data Analysis and Reasoning (for more details of strategies identified byLam et al.[2012] see Chapter5). For each design choice or evaluation strategy, sample methodologies are outlined (presented in a white cloud).
92
In the green box on the map under the visual encoding section, there are a list of design choices that needed to be made. Selection of approaches could be made either through controlled experiments comparing choices, or relevant information found in previously defined guidelines, research or design principles. Questions regarding the visual encoding of the tool that are suited to evaluation by user performance are divided into understanding the limits of human visual percep-tion and cognipercep-tion, or involving a comparison of interacpercep-tion or visual encoding techniques. These questions can be answered by experimentation if no previous research exists. Additionally, visual encoding of the tool may benefit from eval-uation of user experience, where questions are answered through subjective user questionnaires or the process of heuristic evaluation.
In the red box section describing research questions for the interactive tool, there is a second set of user experience questions which may be answered by heuristic evaluation or laboratory questionnaire. An evaluate visual data analysis and reasoning section defines research questions related to decision making and knowledge discovery, where responses can be gathered from case studies or user experiments with the interactive tool.
Ultimately, the creation of this map was a highly worthwhile experience as it allowed us to select the most appropriate evaluation methodologies for the research questions in which we were interested.