Evaluating  Mobile  Applications

Requirements’ gathering is usually the starting point when designing a system. However, evaluating prototype designs adds to the understanding of the target users and their interaction with the application, leading to an iterative process of editing the design. Bowser et al. (2013) found that treating participants, of an evaluation study as co-designers, would enable them to criticise the design more freely.

When it comes to evaluating a mobile app, the context of use plays a significant role which must not be ignored (Savio and Braiterman, 2007; de Sá et al., 2008; Cherubini and Oliver, 2009; Harrison et al., 2013). Many researchers in the field of usability evaluation favour conducting evaluations in the field rather than isolated laboratories (Tsiaousis and Giaglis, 2010; Korn and Zander, 2010; Larsen et al., 2011). They argue that many of the contextual factors that influence the user’s performance such as noise, interruptions, multitasking and lighting conditions are not available in laboratory settings. Thus, many usability problems may not otherwise be discovered. Lab evaluations will discover interface issues and problems;

however, it will not reveal issues relating to the actual use of the app and these may be missed (Larsen et al., 2011, Sun and May, 2013).

Kaikkonen et al. (2005) conducted a study to understand if there were any differences in identifying usability problems when testing in a laboratory or in the field. They found that the same number of problems was identified in both settings. However, Kaikkonen et al. (2005) emphasise the need for more studies to validate their results. Moreover, they argue that laboratory evaluation of some applications that provide location information in particular may miss some usability aspects. Tsiaousis and Giaglis (2010) conducted a study of 64 mobile website participants and found that the lighting, the proximity of nearby people, the motion of nearby objects/people and the environmental sounds significantly affected the effectiveness and efficiency in the using of the mobile website. Lemmela et al. (2008) conducted mobile evaluations of a messaging application for two contexts (in a car and walking) that concluded that context has an influence on the user's preferences in the usage of modalities and interaction strategies and backs up the argument that conducting evaluations in-situ does help to identify a wider range of issues, as context influences usage. Korn and Zander’s (2010) walkshop study, in which they assess the usability by walking in-situ, found a number of usability issues that were not discovered in the lab. These are related to ‘data input under stress’ as well as receiving interesting discussion and reflection from the participant which they think is a result of interacting with the environment.

A recent study by Sun and May (2013), supporting an earlier study by Kaikkonen (2005), found similar numbers of usability issues in both lab and field evaluations. However, there were essential differences in the issues

identified between both the settings. The issues discovered in the lab were mostly interface problems, while issues in the field were mainly about the mobile ‘use’ and were influenced by environment. Participants in the lab were less engaged, and participants in the field were found to be more critical.

However, they concluded that the field evaluation was difficult to control, as the event distracted some participants, forgetting they were taking part in the study and needing prompting.

The context model constructed by Savio and Braiterman (2007) shown in 3.2 gives a good framework when planning the evaluation of the mobile application. It shows the different elements that could have an effect on the usability of a mobile system.

However, in the case of this research, the evaluation takes two forms:

1. Evaluating the learning experience.

2. Evaluating the design, usability, and user experience of the mobile application.

The deployment and evaluation of mobile learning interventions is a growing research area, particularly in higher education. Researchers are still investigating best practice. Evaluating the effectiveness of mobile location-based learning is fraught with difficulties, as discussed earlier in 3.1.5. In order to ensure a rigorous approach to this research, current approaches for evaluation were investigated.

Vavoula and Sharples (2009) defined a now well-established approach (Ahmed & Parsons, 2012) for evaluating mobile learning. They developed a three-level framework for evaluating mobile learning:

1. The Micro level examines two aspects: the individual activities of the users (learners) and the usability of the technology used.

2. The Meso level examines two aspects: the whole learning experience and how well the learning experience links with other activities.

3. The Macro level examines the impact of the new innovation on (a) the established teaching and learning practice and (b) on the educational institution.

Their framework is shown below:

Figure 7 Three level evaluation framework (Source: Vavoula and Sharples, 2009, Copyright, IGI Global. Reprinted with permission of the publisher)

When this current research first began, the Macro level evaluating the long-term impact of the new innovation on the established teaching and learning practice and institutions (Vavoula and Sharples, 2009) was considered to be beyond its scope. However, it was decided later that the Macro level could be incorporated in this research by deployment of the contextual learning model in several modules. This is discussed in detail later in chapter four.

Other evaluation frameworks were investigated (Economides and Nikolaou, 2008; Taylor, 2004; Traxler & Kukulska-Hulme, 2005). Taylor’s (2004) evaluation framework is a task-centred approach concentrating on

‘pedagogical soundness’ to evaluate mobile learning against the user’s goals.

Economides and Nikolaou (2008) have developed a three-area evaluation framework (Usability, Technical, Functional) for evaluating handheld devices for mobile learning. However, this framework is only interested in the characteristics of the handheld device. It does not relate to any pedagogical aspects and, thus, is not an appropriate framework for evaluating the whole mobile learning experience. Furthermore, the strength of mobile learning occurs in the ability of the learners to use their own handheld devices in learning; hence, many aspects of this framework are not appropriate.