METHOD OF THE EVALUATION FRAMEWORK

This section presents the identified methods of evaluating smartphones’ wellness apps that resulted from analysing the available literature by the qualitative content analysis (see section 3.6.2 and 3.7.1).

As previously mentioned in chapter two (section 2.4 and 2.4.2), the two main methods for evaluating wellness smartphones’ apps have been identified. Firstly, Harrison et al. (2013) have evaluated apps according to the PACMAD (People At the Centre of Mobile Application Development) model. This model evaluates the following elements: effectiveness, efficiency, satisfaction, learnability, memorability, errors, and cognitive loads. In addition, it considers three main factors: user, task, and context (Harrison et al. 2013). Each of these elements has its own utility assessed. The PACMAD model is a comprehensive model, which can evaluate apps based on these several general elements. According to Harrison et al. (2013),

“PACMAD model brings together significant attributes from different usability models in order to create a more comprehensive model” (Harrison et al, 2013, p.1).

The authors claim that although none of the elements of the PACMAD model is new, the existing prominently available usability models disregard some of these elements. This leads to an inaccurate evaluation of smartphones’ apps usability (Harrison et al. 2013). However, evaluating large number of apps using the PACMAD model requires extensive resources including time and participants. Thus, apps cannot be evaluated using the PACMAD model within this research project. To illustrate how this model requires extensive resources, consider effectiveness, one of the usability elements of PACMAD model. According to Harrison et al. (2013),

“effectiveness is measured by evaluating whether or not participants can complete a set of specified tasks” (Harrison et al, 2013. p.4).

As this research aims (see Chapter 3, section 3.2.1) to identify the most popular weight loss and diet apps according to specific criteria and evaluation framework, it would be time consuming to evaluate the effectiveness of each app in the aforementioned way. This research has identified 51 Google Play and iTunes apps (see

chapter, section 4.2, 4.3 and 4.4) which is a large number of apps to be individually evaluated.

Some of the PACMAD model elements need extensive time to evaluate such as the learnability element. According to Harrison et al. (2013),

“In order to measure Learnability, researchers may look at the performance of participants during a series of tasks, and measure how long it takes these participants to reach a pre-specified level of proficiency” (Harrison et al, 2013, P.4).

Measuring the learnability element for each of the 51 apps would be time and resource consuming which exceeds the available resources for this research.

Some of the PACMAD model elements are difficult to evaluate such as measuring memorability of an app.

“There is difficulty associated with evaluating Memorability” (Harrison et al. 2013, p.10).

Measuring memorability requires examining participants’ use of apps after a period

of inactivity with it. There is a real issue in recruiting participants who are willing to

return a multiple of times to participate in an evaluation (Harrison et al. 2013). Thus,

evaluating a large number of apps would be time and resource intensive and so exceed the available resources for this research.

By analysing the literature, another method of evaluating wellness apps was identified. There are several studies such as Breton et al. (2011), Azar et al. (2013), IMS (2013), and Abroms et al. (2013) that have evaluated wellness apps by firstly identifying predefined usability elements. For example, in Breton et al. (2011), Azar et al. (2013), the elements were related to weight loss and diet. Then, the apps were evaluated according to the presence or absence of these usability elements in the apps. However, in the study of Abroms et al. (2013), the elements were related to smoking cessation. Each of these studies has developed its scoring system for evaluating apps. To illustrate, Breton et al. (2011) examined the contents of each apps based on the 13 pre-defined weight loss related elements (see Chapter 2, section 2.4.3). If the elements were present in the app, the app then would take a certain score. Then, the app scores of

all 13 elements are calculated and the total score is obtained as an index score for the app. By evaluating several apps based on specified usability elements, each app should have its own index score. The apps can be then ranked according to their index scores.

The presence and absence of the elements could be represented in different ways. It could be represented as numbers such as in the studies of IMS Institute for healthcare informatics (2013) and Abroms et al. (2013). For instance, in Abroms et al. study each item was coded as 0 which indicates, “not present at all”, 1 indicates “partially present”, or 2 which indicates that the element is “fully present”. It could also be coded as yes=1 or no=2 as in the Azar et al. (2013) study or by denoting “X” which means x=1 as in Breton et al.s’ (2011) study.

The second way of evaluating apps is more appropriate for this study as the elements of PACMAD model are general in its nature; hence it might be less efficient if applied to determine the efficacy of weight loss and diet apps. In addition, as previously discussed in this chapter (section 5.2), applying the PACMAD model for large number of apps exceeds the resources available to this research. Thus, Breton et al. (2011) method of evaluating apps was followed here.

Section 5.2 has presented the available methods of evaluating wellness apps in the recent available literature. The next section will discuss the elements of the evaluation frameworks.