5 Discussion
5.1 Revisiting the Research Questions and Contributions
5.1.3 RQ3 How can the utilization of usage data logging be supported in product
experiences from UX evaluation methods and conducting long-term UX evaluations with these methods. For example, experiences from Study I (P3) and II (P4) were disseminated to the collaborating companies as a part of a Long-Term UX Evaluation Toolbox, a document combining information from literature and experiences regarding the methods utilized during the project. Finally, the DrawUX survey tool was presented as a practical contribution that can
potentially support UX evaluation work, having novelty value in its features that support a)
respondents in reporting their experiences over time in remote studies and b) evaluators in analyzing and reporting the results from the experience curve drawing tasks. Online evaluation tools supporting self-reporting, such as DrawUX, might help product development teams, especially UX designers, in conducting remote and face-to-face long-term UX evaluation studies of their products and reporting the findings to relevant stakeholders.
5.1.3 RQ3. How can the utilization of usage data logging be supported in
product development?
RQ3 is “How can the utilization of usage data logging be supported in product development?” RQ3 was approached from two viewpoints: 1) inspecting the requirements and
expected benefits of usage data logging in a specific product development context and 2) supporting the utilization of usage data logging through collaborative development of visual data analytics tools for logged usage data. The main contribution lies in the study’s context, the development of manufacturing systems, as little previous research work is available where visual data analytics tools have been utilized for analyzing logged usage data with industrial manufacturing or related industrial systems.
Some of the identified or expected benefits of usage data logging, as presented in Section 4.3.1, could well apply outside the studied contexts that included developing mobile learning services and systems for manufacturing automation industry. Examples of such benefits were that a) logged usage data can be used to verify users’ own recollections of product usage, b) data logging
does not interrupt normal product usage, c) log data can reveal interesting usage patterns and possibly justify a need for further qualitative inspection, and d) log data can inform developers on how the product is used after major software updates, therefore supporting product development decisions. Indeed, Grimes et al. (2007) also emphasized that data logging does not disturb the user and therefore provides unbiased observational data when studying query logs for search engines. In another example from game development field, Hullet et al. (2012) showed how analysis of long-term logged usage data resulted in recommendations for changes in the user interface of the studied auto racing game. These earlier results from other industrial domains support the generalizability of the identified and expected benefits of usage data logging from this research work. However, the expected business opportunities from usage data logging were more specific to the domain of manufacturing automation. These opportunities included improving customer support services (e.g., fault diagnosis), new customer training offers, providing additional value from customer reports and obtaining evidence for accidents, for example, regarding the liability of the damage. Overall, these results contribute to the current
understanding of how product development practitioners, especially in the manufacturing industry, perceive the benefits of usage data logging. Further research should explore how
successfully log data can meet these expectations and provide additional value for practitioners in product development and other stakeholders, including customers and end-users.
From a product development perspective, logged usage data could be a useful channel especially for development teams that have little opportunities in observing how their products are used in the field. For instance, in the case of supplier companies in the manufacturing industry, customers can be located around the world, making site visits costly. Remotely collected log data can provide an overall view of how the product is used and how individual use patterns emerge, as long as data collection and analysis on individual level is agreed with the customer. However, as noted by Grimes et al. (2007), log data does not explain why the user has made specific choices. Therefore, it seems that while usage data logging might support the generation of UX insights related to practical usability aspects, such as efficiency or effectiveness, insights related to emotional aspects of product use would require other evaluation methods. As results from Study
V suggest, data logging could be useful in identifying and justifying situations where a more
qualitative approach, such as user observation or interview, is necessary to understand reasons for user actions. However, there are ethical and legal issues that need to be considered in this approach, such as the anonymity of the log data and if designers can contact the specific person experiencing problems with the system, based on his or her logged behavior. Often the case can be that log data is anonymized and there is no way to contact specific users to learn more about their experiences with the system. Long-term studies, where participants agree that their product usage is logged over time, could be one potential approach where logged usage data could be utilized together with more qualitative UX evaluation methods. Long-term participation might also decrease the possible Hawthorne effect, i.e. the change in behavior due the feeling of being observed, in comparison to short-term UX evaluations.
According to the findings from Study IV and V, when utilizing self-reporting UX evaluation methods, such as surveys, usage data logging can provide a more realistic view on how the evaluated product is used. Therefore, it seems that exploring logged usage data can be beneficial in both longitudinal (Study V) and retrospective (Study IV) UX evaluation studies, assuming that log data is available from the evaluation period. An interesting approach could be to study the memories of experiences with retrospective curve drawing tools such as iScale, UX Curve or DrawUX, and compare this data with logged usage data to identify use frequencies, patterns in use and their relation to the memories of experiences. In retrospective studies, one useful approach is to explore the visualized log data together with users, as it can support the recalling of events. Bhavnani et al. (2017) utilized log data visualizations in retrospective interviews regarding mobile phone usage and found that log data provided helpful cues for the participants to recall details from app usage. In their conclusion, Bhavnani et al. emphasize the need for ethical considerations of finding the balance between what activities are logged and what details of the log data are presented to the participants, while not making participants uncomfortable with the logging. This would be an interesting question to study also in the work context, such as with users of flexible manufacturing systems (Study V). What level of detail in the log data users are comfortably willing to share and discuss with designers for product development purposes? Supposedly, in work context users might perceive such data logging more negatively, feeling that their personal work performance and skills are evaluated, and that if they use the system in a “wrong way”, it may affect their reputation in the workplace. However, further investigation is required in this topic.
In the results Section 4.3.1, as a practical contribution, a set of questions is provided to inspire discussion among development team members to consider the feasibility of usage data logging in their work context. The proposed topics include 1) possibilities and goals for usage data logging, 2) skills required for data analysis, 3) data access and secure handling of the data, 4) tools for data analytics and visualization, and 5) transferring the “raw” data to a suitable form for data analytics tools (i.e., requirements for data wrangling). Although not an exhaustive list of all topics related to usage data logging, these questions can be used by development teams to inspire
discussions on the key aspects related to the utilization of usage data logging. For instance,
the ethical and legal considerations regarding usage data logging should be discussed in the first (possibilities and goals) and third (data access) points, as they may greatly restrict what data can be logged and utilized e.g. for product development purposes.
Guidelines to support the development of visual data analytics tools for logged usage data are presented in Section 4.3.2. While these guidelines were derived from a case study in the domain of flexible manufacturing systems (P6), some of the guidelines reflect experiences reported in studies from other domains, such as game development (Medler et al. 2011). For instance, Medler et al. (2011) also emphasized that gathering an interdisciplinary team can greatly support the development of analytics tools. Furthermore, Sedlmair et al. (2012) also proposed that accessing real logged usage data is one major design study pitfall. The proposed guidelines also agree with earlier research, which has suggested that data analytics tools should a) support users who are less
familiar with analytics (Heer & Kandel, 2012) and b) provide support for collaboration (Shneiderman et al. 2006). It is possible that the proposed guidelines could be applied outside the manufacturing industry to support the development of visual data analytics tools, such as in the development of different web-based services or industrial systems. Furthermore, while in Study
V an external team of researchers from academia developed the data analytics tool, the proposed
guidelines could inform also internal company teams developing analytics tools for logged usage data.
Although there can be various challenges in the collaborative development of data analytics tools between researchers from academia and practitioners from industry, as summarized in the guidelines from P6, we argue that this can be a viable approach to supporting product development teams in utilizing logged usage data. Participation in the development and evaluation of analytics tools can benefit the collaborating company by improving the basic data literacy skills of the employees (Medler et al. 2011). Medler et al. (2011) argue for developing visual data analytics tools in parallel with product development. This is advisable whenever possible, as it can decrease the need for data wrangling when the data logging services in the system and transferring the data to the analytics tool can be designed in parallel. Indeed, one of the challenges in Study V was that the data analytics tool was developed by an external research team after the data logging services had been implemented. This related to the fourth guideline: “Allocate Resources to Explore the Log Data Structure Prior to Data Wrangling” (section 4.3.2). In our case, analytics tool developers faced challenges in transferring the log data to a suitable format for the analytics tool and had to work in close collaboration with the system developers who had a deeper understanding of the logging process of the manufacturing system.
To conclude, the proposed guidance for developing visual data analytics tools for usage
data logging contributes to filling the gap in explorative research done in manufacturing automation domain. By providing support for the development of visual data analytics tools for
usage data logging, product development teams should have better tools at their disposal for utilizing usage data logging to support their work. Finally, from UX design and evaluation perspective, usage data logging is considered to be most valuable approach in supporting other UX evaluation methods, that can provide more qualitative data e.g. to explain reasons for users’ actions and experiences with the product.