Development and Evaluation of Visual Data Analytics Tools with Users

This section provides a summary of the related work that discusses the development of visual data analytics tools together with users. The collaborative development of visual data analytics tools is an approach utilized in Study V (section 3.2.5) and one of the contributions from this study aims to support similar development activities. However, the aim here is not to provide a list of currently available data analytic and visualization tools and approaches, but to review the previous work done in presenting design approaches, guidelines and relevant case studies where visual data analytics tools were developed together with users. Finally, previous studies done in manufacturing automation context are highlighted, as related to research done for this thesis.

In this work, the visual data analytics tool refers to software applications and web services that offer a variety of data analytics and data visualization features for inspecting log data, with the aim to provide users with insights regarding the inspected data and what it represents. According to categorization done by Hilbert & Redmiles (2000), visual data analytics tools that support the inspection of logged usage data belong to the techniques of integrated support, offering a variety of transformation, analysis and visualization techniques (see Table 2).

There are various proposed approaches and guidelines for supporting the development and evaluation of visual data analytics tools with users. Carpendale (2008) presents a good overview of different evaluation approaches and methods for visualizations, while Munzner (2009) provides advice on when to choose between different methods. The choice of an evaluation approach depends on the goals of the study. For example, quantitative laboratory experiments focus on precision, while sample surveys can aim for more generalizable results (Carpendale, 2008). Utilization of qualitative evaluation methods such as the observations or interviews of users, especially during field studies in the real use context, can support evaluators in obtaining a richer understanding of different factors that may influence the development and usage of visual data analytics tools (Carpendale, 2008; Patton, 2001). Carpendale (2008) encouraged that more

studies evaluating visualizations should utilize qualitative methods.

The Multi-dimensional In-depth Long-term Case Study (MILC) (Shneiderman & Plaisant, 2006) is an approach to evaluate visualization tools with both qualitative and quantitative methods, originating from the studies of creativity support tools (Shneiderman et al. 2006). MILC combines field studies with participant observation, interviews, surveys and automated logging of user activity. Shneiderman & Plaisant (2006) suggest that MILCs can be beneficial for studying the

efficacy of novel visualization tools regarding their strengths and for iterating the tool with end- users while providing evidence to warrant further development. The MILC approach and its derivatives (Perer & Shneiderman, 2008) have been utilized to support the development and evaluation of visualization tools for event sequence analysis (Wongsuphasawat et al. 2011) and electronic medical records analysis (Stolper et al. 2014). MILC has been identified as a relevant approach to evaluating how domain experts utilize visual analytics over time (Gotz & Stavropoulos, 2014). Wongsuphasawat and colleagues (2011) found that periodic meetings with a domain expert supported the generation of insights and allowed additional questions and guidance for tool development. Stolper et al. (2014) also demonstrated benefits in utilizing case study approach when documenting insights that were generated during the long-term use of a visual data analytics tool.

Lam et al. (2012) stress that approach to evaluating visualizations should be based on evaluation goals and questions rather than methods. They approach the topic by presenting seven types of evaluation scenarios for visual data analytics tools, based on the literature review of 850 papers from the information visualization domain. The scenarios are categorized into those that can support the understanding of data analysis processes and those that can help the evaluation of the visualizations themselves. One of the presented scenarios is the user experience evaluation of visualizations, where the proposed goal is to understand “what do my target users think of the

visualization?” (Lam et al. 2012). In addition, Lam et al. (2012) propose the following questions

to be considered in UX evaluations of visualization tools:  What features are seen as useful?

 What features are missing?

 How can features be reworked to improve the supported work processes?  Are there limitations of the current system which would hinder its adoption?  Is the tool understandable and can it be learned?

Sedlmair et al. (2012) present a nine-stage framework for conducting design studies and practical guidance for designing visualization systems in collaboration with domain experts. Alongside the framework, they describe 32 design study pitfalls to guide the whole process from learning and designing to reporting design studies. The framework is based on their own experiences and literature review in the fields of human-computer interaction (HCI) and social science. Recently, Crisan et al. (2016) extended the framework by providing practical guidelines that consider also external constraints that can affect visualization design and evaluation. By taking into account the external constraints, regulatory and organizational, visualization researchers and practitioners should be able to improve their visualization solutions regarding their utility and validity, while also improving the likelihood that collaboration with industrial partners is successful.

Several researchers have reported experiences from conducting visualization evaluation in specific work contexts, such as large company setting (Sedlmair et al. 2011), bioinformatics (Saraiya et al. 2006) and game development (Medler et al. 2011). Sedlmair et al. (2011) listed

challenges and provided recommendations for evaluating data analysis processes and visualization tools in a large company setting, based on their experiences from a variety of studies in this context. Saraiya et al. (2006) conducted a long-term study with bioinformaticians to evaluate how they use visualizations to gain insights into the data. They emphasize two reasons for conducting long-term evaluation studies in real context instead of short-term laboratory experiments: 1) to recognize the users’ natural motivation to do data analysis and 2) the evaluation of the significance of insights. Saraiya et al. (2006) conclude that longitudinal studies make it possible to inspect the long-term insight generation process and identify any long-term usability problems with data visualization tools. Medler et al. (2011) developed a visual game analytics tool Data Cracker in collaboration with a game development team who were the target users of the tool. Medler et al. (2011) argue for developing visual data analytics tools in parallel with product development. They propose that visual prototypes are used when discussing how the tool could be beneficial for the development team. Tool designers should keep in mind the broad audience of end-users and encourage users to utilize the tool also after the product release, to provide useful feedback on the development of future products.

To the candidate’s best knowledge, few scientific publications exist where visual data analytics tools have been utilized especially for analyzing usage data with industrial manufacturing or related industrial systems. Holzmann et al. (2014) studied how user interaction data from a touch screen based robot controller can be the acquired and visualized, in order to provide cost-efficient solutions for evaluating the usability of handheld terminals in the automation industry. The goal was to support developers by inspecting how users interact with the user interface and by identifying possible issues with the users’ workflow, such as navigation problems or unused functions. Based on interviews with a programmer and two project managers from automation industry enterprises, Holzmann et al. (2014) identified navigation path analysis and usage intensity as the most important topics for data logging in this context. In another study on automation industry context, Grossauer et al. (2015) developed a prototype for visualizing navigation flows through an application. After applying the visualization tool to multiple datasets, they suggest that developers of such tools should provide users 1) a wide variety of filters and 2) views that show the whole navigation data and allow the inspection of individual sequences.

In summary, there is little public research available on the subject of usage data logging in manufacturing automation context, especially regarding how the developers and designers of automated manufacturing systems utilize logged usage data. First, more information is needed

about the expectations, benefits and challenges related to usage data logging in manufacturing automation and related industrial contexts. Second, in order to support the utilization of logged usage data with visual data analytics tools that provide positive user experience, guidance to support the development of analytics tools in manufacturing automation context is required.