Publications describing EUD-ESM tools generally lack evaluation of both the client-side mobile application, and in particular, the EUD-ESM tool itself. The categories of research purposes proposed by Kjeldskov and Graham [17] and described in Section 1.3, are repeated as follows. • Understanding:Research aimed at understanding the particulars of a phenomenon studied • Engineering:Research aimed at the original development of a tool or technology
• Re-engineering: Research aimed at the engineering of modifications or extensions to an existing tool or technology
• Evaluating:Research aimed at the assessment, validation and assurance of tools, technology, models and frameworks
• Describing:Research aimed at describing the ideal properties of a system or situation it is notable that the majority of EUD-ESM research is exclusive to theEngineeringcategory. In such research, the development of an EUD-ESM tool is described, but with no formal evaluation. It was expected that more systems would additionally be included in theEvaluatingcategory, given that 46% of the surveyed EUD publications in [18] which describe a tool also follow up with a lab evaluation. In fact, only two publications report a formal usability evaluation of their described EUD-ESM tools [101, 153].
Given the lack of evaluation research, it is unsurprising that only the work of Ludwig et al. [101] could be consideredDescribing research by Kjeldskov and Graham. A publication with the purpose of describing ideal system properties requires supporting evidence of what properties are actually beneficial, based on empirical evaluation. Although the surveyed publications often state that informal usability testing has been conducted with researchers, these claims do not advance the field in terms of best practice for how to model an ESM study in such a way as to be understood, and thus created and modified, by non-programmers.
The concept of usability has been defined in a variety of ways, often in terms of the ISO standard, which states that usability is“the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”[175, p. 537]. Usability is thus largely context-dependent, and also depends on the measures used to assess this effectiveness, efficiency and satisfaction. Hornbæk provides a comprehensive review of the methods used to measure these three constructs [176]. Different research methods were also undertaken within the studies in this chapter. In the first study, quantitative measures such as task time and error rate were taken. Conversely, the final study did not quantify any such measures for statistical analyses, but instead took a qualitative approach by obtaining insight from participants’ think-aloud during task completion and retrospective semi-structured interviews.
6.1. RESEARCH METHODS 127
Table 6.1:The three usability studies of this chapter in relation to the usability constructs employed
Study #1 Study #2 Study #3 Effectiveness Accuracy of solutionTask completion
Errors made
Task completion Accuracy of solution
Task completion Accuracy of solution
Efficiency Task completion time
Satisfaction System Usability Scale
Post-study interview System Usability Scale Post-study interview The three studies in this chapter are as follows: a study with 20 participants where quantitative metrics on task time and error rate were collected (Study #1, Section 6.2) a smaller study with six health psychology students (Study #2, Section 6.3) and a final study with 10 participants that collected more qualitative data (Study #3, Section 6.4). The studies are summarised in Table 6.1 with respect to the research methods, relevant to each of the three usability constructs, that they employ.
6.1.1 How important is usability?
As discussed in Section 4.1, evaluations of EUD tools are most frequently conducted within the constraints of lab-based usability studies. However, such studies are limited in their capacity to determine the acceptance of an EUD tool in its intended context of use. As highlighted in [143], many products are commercial successes despite being difficult to use, and conversely, highly usableproducts may fail if they are notusefulin practice.
Nevertheless, in the case of an EUD tool, usability studies are necessary. Thoroughly researched models of technology acceptance all consider ease-of-use as a key prerequisite for the adoption of a new technology, particularly in its early stages. For example, “perceived ease-of-use” is a core construct of the Technology Acceptance Model [145]. The Unified Theory of Acceptance and Use of Technology has the similar construct of “effort expectancy” [146]. The purpose of EUD is to makeusefulsoftware functionality accessible to its end-users. Thus, if usability issues are not addressed in an EUD tool, then these potentially useful features are still inaccessible to end-users, who are likely to abandon unusable technology in favour of delegating development to a professional.
This chapter thus describes three formal usability studies of Jeeves conducted during its iterative development. In doing so, insights into the benefits and challenges of the blocks- based programming paradigm were obtained, as well as preliminary feedback on the real-world utility of such a tool. In answering the research sub-question “How can the development effort
128 CHAPTER 6. USABILITY ANALYSIS
be reduced to allow rapid creation of experience sampling apps with no programming?” these studies aim to evaluate the implementation decisions discussed in the previous chapter.