User Experience Assessment Methods

User experience assessment methods play a vital role in the product development process to ensure the achievement of the product’s goals (Roto, Obrist, and Väänänen-Vainio-Mattila, 2010). The assessment or evaluation of user experience systematically improves the product and, thus, the experience (Ketola and Roto, 2009; Väänänen-Vainio-Mattila, et al., 2008a).

In order to establish a systematic procedure for evaluating user experience, Roto et al. (2009) outline four steps that need to be performed: (1) obtaining a good understanding of user experience, (2) defining user experience metrics, (3) developing user experience evaluation methods based on these metrics, and (4) integrating user experience evaluations into the product development process.

Additionally, in the product development setting, user experience evaluation should attend to the causal relationship between the experience and the product; that is, to how the examined product influences the measured experience

(Väänänen-Vainio-Mattila, et al., 2008b). This is to help in identifying the product’s elements that need to be modified in order to improve the whole experience.

The requirements for user experience evaluation methods in the industrial context are partially different from those that apply in the academic context. In industry, the main requirements for the methods are practicality, utility, and simplicity. In contrast, the academic context focuses more on the scientific rigour of the methods.

Fundamentally, as asserted by Law (2011), user experience measures need to be meaningful, valid, and useful. As the result of the UXEM workshop, Väänänen-Vainio-Mattila et al. (2008b) summarised and listed the requirements for practical methods of user experience evaluation in product development:

• Fast, lightweight (i.e. requiring fewer resources), and cost-efficient for quick iterative evaluations.

• Valid, reliable, and repeatable.

• Requiring a low level of expertise to apply.

• Applicable to different types of products.

• Applicable to concept ideas, prototypes, and final products (to assist in understanding how user experience develops through the stages of product development).

• Generating comparable outcomes (both qualitative and quantitative).

• Useful to different departments and multi-disciplinary teams in a company.

• Suitable for various phases of product lifecycle.

• Suitable for different target user groups.

It is stressed that user experience evaluation is significantly different from usability evaluation. While the latter focuses on task completion and performance measures (efficiency and effectiveness), the former also places emphasis on hedonic aspects of product use, and on users’ feelings, motivations, and expectations with respect to using a designed product in various contexts (Obrist, et al., 2009; Roto, Obrist, et al., 2010). Consequently, objective measures, such as number of errors and task completion time, are not accurate predictors of user experience. Rather, subjective measures (e.g. emotions and judgments), which take contexts of use, temporal aspects, and other non-instrumental factors into account, should be included. For example, by highlighting the dynamic and situated characteristics of user experience,

Vermeeren and Kort (as cited in Pals, et al., 2008; Vermeeren, et al., 2008) recommend that user experience assessment instruments should include:

• Collection of both quantitative data (what users do) and qualitative data (what users think and experience), using both objective and subjective techniques.

• Unobtrusive capture of data in order not to influence the user experience.

• Enabling of continuous or timed measurements and longitudinal studies.

• Support for contextual or in situ measurements to obtain information that reflects users’ experiences in their actual everyday lives.

In response to the needs for user experience evaluation, a multitude of methods and techniques have been proposed. Many of them have their roots in traditional usability methods, but are extended to capture hedonic, emotional, and experiential aspects of user-product interactions. They cover, for example, attitude scales, co-discovery, focus groups, think aloud protocols, reaction checklists, field observations, interviews, immersion, expert appraisal, property checklists (Jordan, 1999, 2000), experience diaries (Karapanos, et al., 2009), and questionnaires (Hassenzahl, 2004;

Laugwitz, et al., 2008; Thayer and Dugan, 2009).

More recent methods include visual techniques (Forlizzi, Gemperle, and DiSalvo, 2003; McDonagh, Bruseberg, and Haslam, 2002; Sleeswijk Visser, Stappers, van der Lugt, and Sanders, 2005), emotion measurements (Desmet, 2003a; Desmet and Dijkhuis, 2003), facial expression assessment (Hazlett and Benedek, 2007; Kaiser and Wehrle, 2001), and psycho-physiological techniques (Mahlke and Thüring, 2007; Mandryk, et al., 2006).

Other methods for investigating user experience include experience sampling or experience diary using electronic devices (Intille, Rondoni, K ukla, Ancona, and Bao, 2003; Isomursu, et al., 2004; Swallow, et al., 2005), narrations or storytelling (Geven, Schrammel, and Tscheligi, 2006; Korhonen, et al., 2010b; Schrammel, et al., 2008), and retrospective sketching for evaluating the dynamic of user experience over time (Karapanos, et al., 2010; Kujala, et al., 2011). Roto, Lee et al. (2010) created a comprehensive list of 82 user experience evaluation methods along with their descriptions. This list is available at http://www.allaboutux.org/all- methods.

With a large number of user experience evaluation methods available, the selection of appropriate methods for particular needs or purposes becomes crucial (Law, 2011). The methods’ categorisation assists designers and researchers in this regard.

Roto, Obrist et al. (2010) have categorised the existing methods into seven groups:

(1) lab studies with individuals, (2) lab studies with groups, (3) short field studies, (4) longitudinal field studies, (5) surveys, (6) expert evaluations, and (7) mixed methods.

Vermeeren et al. (2010) have built on this categorisation by classifying the methods into a number of main categories (e.g. type of collected data, information source, period of experience, and location), each of which consists of several sub-categories.

To illustrate, the user experience measurement conducted by Mahlke and Thüring (2007) belongs to the ‘lab studies with individuals’ and ‘mixed methods’ categories.

Based on their user experience framework (Section 2.4.2), Mahlke and Thüring (2007) employed a combination of methods to assess all user experience components and their interrelationships. The methods collected users’ subjective information (via questionnaires), physiological responses, facial expressions, and behavioural data.

Several computer-simulated portable digital audio players, differing in their usability and visual aesthetics, were used as stimuli. The authors discovered that usability (instrumental) and aesthetics (non- instrumental) factors affected users’ emotional reactions, and that usability had a greater impact on both users’ emotions and their overall judgments. They also found that visual aesthetics had no significant impact on emotional reactions in terms of motor expressions and physiological responses.

Another example is the context-aware experience sampling tool (Intille, et al., 2003), which belongs to the ‘short field studies’ or ‘longitudinal field studies’ categories.

This method enables researchers to perform experience sampling studies using mobile computing devices, and to gather data from users in specific situations that are detected by sensors. Intille et al. (2003) specify that the tool allows data sampling directly through questioning the users, and indirectly through the carried sensor devices (e.g. by automatically recording the user’s location via GPS to provide data about the context of use). This method can be useful for longitudinal user experience studies, since the mobile device can be attached to the study participants for days or weeks, and the changes in context can be recorded from time to time. Thus, the

dynamic or temporal aspects of user experience can be effectively captured, especially when evaluating user experience after product purchase.

The two methods explained above assess users’ experience while or after they use the product or system, as do most others that have been proposed. Only a small percentage of the existing methods can be utilised to evaluate user experience before usage (Bargas-Avila and Hornbæk, 2012; Vermeeren, et al., 2010). In line with the specific theme of this research – that is, anticipated user experience and early assessment of user experience – the next section focuses on discussing the methods and approaches that are intended for assessing user experience in the period before interactions, and in the early stages of the design process.

3.2 USER EXPERIENCE ASSESSMENT IN THE EAR LY PHASES OF