Long-Term UX Evaluation Methods

HCI literature offers a vast number of available methods for UX evaluation. For example, Vermeeren et al. (2010) collected and analyzed 96 UX evaluation methods from academia and industry to evaluate how they are utilized in HCI field. 34 of the identified methods were reported being able to evaluate the long-term use of products. In another study, Rajeshkumar et al. (2013) created taxonomies for 89 UX evaluation methods. Evaluation methods can help designers choosing the best design, confirming that the design is on right track, or assessing that the designed prototype or the final product meets the UX targets (Vermeeren et al. 2010).

Jain et al. (2010) state that there are no specific methods required for longitudinal studies. Instead, they encourage combining quantitative and qualitative methods. In-situ methods that collect user feedback repeatedly over time, such as diaries (Bolger et al. 2003), Experience

Sampling Method (ESM) (Csikszentmihalyi & Larson, 1987), and Day Reconstruction Method

(DRM) (Kahneman et al. 2004) have been used in longitudinal studies. Retrospective methods such as Change-Oriented analysis of the Relationship between Product and User (CORPUS) (von Wilamowitz-Moellendorff et al. 2006), iScale (Karapanos et al. 2010, 2012b), UX Curve (Kujala et al. 2011), DrawUX (Varsaluoma & Kentta, 2012), and MemoLine (Sim et al. 2016) can be less taxing for participants when compared with repeated measurements. Retrospective methods are prone to biases as they rely on users’ memories of experiences (Kahneman et al. 1993; Schacter, 1999). However, memories can still be important information for product development purposes, since the experiences that customers report to others and the customers’ future behavior can be guided by these memories (Norman, 2009; Karapanos et al. 2012b). Karapanos et al. (2012b) suggest that retrospective techniques can be a viable option in studies where memories have higher importance than actuality. Next, the methods that were utilized in studies related to this thesis are introduced.

iScale. Grounded on the theories of the retrospective reconstruction of experiences and

episodes from memory, Karapanos et al. (2010; 2012b) developed an online survey tool, iScale, to support respondents in recalling their experiences with a product over time, while minimizing the retrospective bias. With iScale, users are first asked to evaluate 1) the product’s evaluated quality (chosen by the evaluator) just before purchasing it and 2) how their opinion has changed since then. Next, users are presented with a timeline from a moment of purchase to present time. Respondents continue by “sketching” a line that consists of linear segments that represent how respondents’ perception of the evaluated quality changed over time. For each segment, the participant can add an experience report describing a cause of change in the evaluation. Thus, iScale provides graphs illustrating the remembered changes in experiences and experience narratives that can explain reasons for these changes. Karapanos et al. (2012b) conclude that in comparison with face-to-face interviewing techniques, a structured process for self-reporting, such as iScale, can survey large samples and therefore also “inquire into rare experiences and

atypical behaviors.” Karapanos et al. (2012b) showed that sketching the experience over time can

increase the amount and the richness of the information recalled when compared to free recall, where no sketching is involved. However, as noted by Kujala et al. (2011), their study did not

provide the interpersonal analysis of the graphs and their trend information, which could have provided information on how the overall evaluation is affected by the chronological order of experiences.

UX Curve. Aiming at the more cost-effective elicitation of longitudinal UX data, Kujala et al.

(2011) created a pen-and-paper based method called UX Curve. In comparison to iScale, Kujala et al. (2011) note that UX Curve is designed to be used in face to face setting with the participant, while iScale is aimed more as an independently used self-reporting tool. UX Curve aims to support respondents in retrospectively reporting their experiences with a product. The method aims to support researchers in understanding the reasons why and how the user’s experience may have changed over time. UX Curve includes a template presenting an empty two-dimensional graph area for drawing a curve and separate lines that are used for explaining the changes in the curve. The horizontal axis on the graph represents time from the moment of purchase until the current moment, while the vertical axis represents the intensity of the evaluated experiential aspect, such as ease of use. A vertical line divides the graph into positive upper and negative lower parts. In a validation study for the UX Curve method, 20 mobile phone users, with 3 to 12 months of usage experience, reported their experiences by drawing experience curves and describing possible changes in their relationship towards the phone (Kujala et al. 2011). Although curve drawing was considered challenging by some respondents, it was also found interesting and interactive, and all participants successfully drew their experience curves. The exact timing of the remembered experiences was considered difficult, suggesting that UX Curve rather provides the approximate reconstructions of the meaningful events. However, Kujala et al. (2011) argue that these events are important for designers as they can help in identifying issues that create positive experiences and affect customer loyalty. When comparing different curve types, Kujala et al. (2011) concluded that Attractiveness curve provided the largest number of reasons for explaining the change in user experience. Furthermore, the improving trend of the Attractiveness curve was related to the willingness of recommending the product to others, which is an important measure when estimating product growth (Reichheld, 2003).

AttrakDiff and AttrakDiff2. AttrakDiff (Hassenzahl et al. 2003) and AttrakDiff2

questionnaires (Hassenzahl, 2004) were designed for measuring the hedonic (identification and stimulation) and pragmatic UX attributes of a product or service. AttrakDiff2 consists of 21 semantic differentials (word-pairs) on a 7-point Likert scale. AttrakDiff is one of the most used questionnaires for measuring UX (Bargas-Avila & Hornbaek, 2011) and provides feedback on how users perceive the product at the current moment. However, for evaluating the change in UX over time, AttrakDiff can be used in repeated measurement and longitudinal studies to allow comparison between different measurement points over product usage. In addition to the UX attributes included in AttrakDiff, studies utilizing it have also been measuring aspects such as attractiveness (e.g. pleasantness or beauty) and overall goodness of a product (e.g. Hassenzahl, 2004; Kujala et al. 2013).

Although various methods and tools are available for evaluating long-term UX, little

of products and services is studied from the perspective of practitioners in companies. In

particular, few empirical studies have focused on the utilization of retrospective “experience

curve” methods such as iScale or UX Curve. Finally, the utilization of long-term UX evaluation

methods together with usage data logging has received little attention in the literature. In the following section, the previous literature in HCI is examined regarding usage data logging and the development of visual data analytics tools with users, with a specific focus on manufacturing automation context.