1.3.5 How can a development tool be used in practice?
Objective 7 - Following usability feedback gathered from the previous objective, conduct interviews with researchers and clinicians addressing the perceived usefulness of Jeeves in their domain of research or practice. Again, this isevaluatingresearch, focused on the real-world utility of Jeeves, throughsurvey researchwith potential end-users.
Objective 8 -To evaluate researchers’ practical application of Jeeves, rather than simply their perceivedutility, conductcase studieswith researchers in which they apply Jeeves in addressing their own research questions. This is furtherevaluatingresearch, demonstrating the utilityand usability of Jeeves in its context of use.
1.4 Contributions
Through the interdisciplinary work described in this thesis, the following theoretical contributions are of value to human-computer interaction, as well as psychology research and clinical practice. • A literature review of user-centred design studies and strategies for maximising ESM study
utility, from which guidelines for future ESM tools are derived (Chapter 3)
• A series of user studies that demonstrate the usability of the Jeeves blocks-based programming paradigm, with implications for domain-specific EUD and blocks-based programming (Chapter 6)
• A qualitative analysis of interviews, observations and case studies involving researchers and clinicians, from which a set of requirements are derived for ESM app EUD, with implications for other novel technology deployed in such domains (Chapters 7 & 8)
• A set of design guidelines for the development of future EUD-ESM tools, consolidated from the results of analytical and empirical research (Chapter 9)
Further, the Jeeves platform represents a practical contribution, which can be utilised by non- programmers interested in running ESM studies. It consists of:
• The Jeeves blocks-based programming tool, allowing non-programmers to specify tailored, context-sensitive ESM apps, collect data and monitor compliance in real-time
• A dynamic Android ESM app (JeevesAndroid) that has been developed in parallel with the visual programming environment that can run any apps created with the Jeeves tool
• A framework that allows apps to be modified in real-time, and implicit and explicit self-report data to be instantly accessible by researchers
10 CHAPTER 1. INTRODUCTION
Table 1.2:Objectives addressed in each of the primary thesis chapters
Chapter Objectives Addressed
2 O3 - Review recent ESM studies in psychology and medicine,compare and contrast with ESM studies in computer science 3/4 O4 - Review state-of-the-art EUD tools and address limitations
5 O5 - Implement Jeeves, JeevesAndroid and the overall framework
6 O6 - Conduct targeted, task-based usability evaluations of Jeeves
7 O1 - Understand working practices of researchers and cliniciansO7 - Evaluate researchers’ and clinicians’ perceived utility of Jeeves 8 O2 - Observe working practices of potential clinician end-usersO8 - Conduct case studies with psychology researchers using Jeeves
1.5 Thesis Overview
This thesis is organised into nine chapters. Table 1.2 describes where the objectives outlined in previous sections are addressed in each of these chapters. As can be seen from this table, the objectives are not addressed in order. Further, the working practices of researchers and clinicians, and associated requirements, were not addressed until after implementation of a fully-working prototype. Justification for this limitation is described in Chapter 9.
Chapter 2provides detailed background information on ESM, explaining the various benefits and challenges of ESM, its use in research as well as clinical practice, and the evolving contribution of technology to ESM. Further, a review of recent ESM studies is presented, demonstrating the current challenges.
Chapter 3 reviews background literature on the use of smartphone apps in healthcare and psychology, deriving guidelines for future ESM tools from their stakeholders’ perspectives. From this, a model is synthesised that represents these features as interactions between apps, researchers and participants.
Chapter 4reviews background literature related to EUD. From this, a review of tools for ESM study creation is presented, analysing each tool with respect to the app features identified in the previous chapter, providing motivation for the implementation of Jeeves.
Chapter 5 presents design requirements of an EUD tool for ESM app creation, based on the literature reviewed in Chapters 3 and 4, and how the design of Jeeves satisfies these requirements. Design decisions of Jeeves are described in relation to established design frameworks and principles. Finally, a description of the full platform is given, including the JeevesAndroid app
1.5. THESIS OVERVIEW 11
and overall architecture.
Chapter 6 describes three studies that were conducted to investigate the usability of Jeeves for non-programmers, and the iterative development improvements that were made as a result of feedback from these studies. Additionally, design guidelines for domain-specific visual programming environments are derived from this study feedback.
Chapter 7 details a qualitative analysis of interviews that assess the potential contribution of Jeeves to psychology research and clinical practice. These interviews discuss current working practices and associated difficulties of researchers and clinicians, and elicit their preliminary feedback on the Jeeves prototype. From these interviews, further factors addressing the utility of a tool for ESM app creation are derived.
Chapter 8 describes case studies that validate the efficacy of Jeeves for creating a study specification and running it with participants, with further feedback on real-world usability and utility. It also describes an observation session that was conducted in a local clinic to determine further factors for adoption in this domain.
Chapter 9, the conclusion, summarises the contribution of this thesis to the field of end-user development, as well as the interdisciplinary contribution to psychology and medicine. This conclusion also postulates future work planned for Jeeves and the limitations of this research.
2
CHAPTER
TWO
EXPERIENCE
SAMPLING
This chapter describes the Experience Sampling Method (ESM) in more detail, including the general methodology, its advantages and disadvantages, and relevant domain-specific concepts. ESM, and the closely related method of Ecological Momentary Assessment, have their roots in psychology and clinical research respectively. In human-computer interaction, the methodology has been borrowed for evaluating how users interact with technology in natural contexts, just as ethnographic methods have been borrowed from social science for the same purpose. As a contribution to the method, human-computer interaction researchers can develop interfaces that facilitate the creation of ESM smartphone apps - the overarching goal of this thesis.
This chapter defines ESM in more detail, describes its benefits and limitations, and how it has contributed to research in a variety of fields. Further, a review of recent ESM literature has been conducted, providing insights of relevance to this thesis.
2.1 Overview
Experience sampling studies involve asking participants to complete assessments on their current thoughts, feelings or context, as they go about their everyday lives. Indeed, the aforementioned Ecological Momentary Assessmentsummarises the defining features of the method in its title: • Ecological:Data collection takes placein situ, as participants go about their everyday lives,
as opposed to an artificial lab environment.
• Momentary:Participants are asked to report their statein-the-moment, rather than summarise over many hours, days or weeks.
• Assessment: Participants provide data through explicit self-report of their current states (although implicit data is also used as a secondary source, as described in Section 3.1).
14 CHAPTER 2. EXPERIENCE SAMPLING
Table 2.1:Summary of main benefits and challenges of experience sampling
Benefits Challenges
Participants are assessed in their natural environments -high ecological validity
Implementation can be difficult, costly and burdensome for researchers and participants Participants are asked about events close to
their occurrence,minimising recall bias
Continuous assessment can influence variables of interest -assessment reactivity
Repeated assessment allows observation of
dynamic and contextual associations
Repeated assessment places high burden on participants, provokingnon-compliance
These features of ESM provide benefits over other qualitative research methods of data collection such as retrospective self-report, or face-to-face interviews. Nevertheless, these benefits also present challenges that require further research and interdisciplinary knowledge to overcome. This section will discuss both these benefits and challenges, and their implications for research, with a summary shown in Table 2.1.
2.1.1 Benefits
The benefits of ESM for capturing the intricacies of daily life as they occur have been widely acknowledged and described in various reviews on the methodology. This section summarises these benefits, which have justified its continued use in psychology and medicine, and its novel application in other fields. The following benefits are agnostic to the technology used, and are thus applicable to paper-based applications of ESM as well as mobile technology. (More specific advantages of smartphones for ESM are detailed further in Section 3.1.)
2.1.1.1 Ecological Validity
The validity of ESM self-reports are enhanced by both the temporal constraints on assessments, as well as the contexts in which these assessments take place. Having participants report on experiences, feelings, or other variables in their natural environment increases the ecological validity of assessments [19]. Conversely, participants in an unfamiliar laboratory or clinic are less likely to disclose sensitive information. In medicine, a relevant phenomenon is that of “white-coat hypertension”, where patients experience raised blood pressure in direct response to
the clinical environment.
The characteristics of a participant’s context, including location, time, or presence of others, may be directly linked to the emergence of specific thoughts or behaviours [14]. Traditional assessments that take place at a lab or clinic fail to accurately capture the context of symptoms,
2.1. OVERVIEW 15
whereas experience sampling measures can be used to acquire this context at the time of assessment. This is necessary for monitoring certain behaviours; for example, substance use is highly dependent on situational cues and social context [20].
2.1.1.2 Minimal Recall Bias
Themomentaryaspect of experience sampling alleviates participants from relying on recollection of past experiences. When asked to recall emotions or physical symptoms from a previous point in time, responses are inherently subjected to “recall bias”. The passage of time can distort one’s perception of a particular incident, and indeed one’s current state can influence recollection of past state. Research has also shown that participants may employ a “peak-end rule”, whereby summaries of experience over time are biased towards the most salient and recent experiences [21].
Experience sampling minimises this bias by reducing the time between an event of interest and the time at which this event is assessed. Further, summaries of experience are largely unnecessary, as only participants’ immediate state is requested at each repeated assessment [13]. Studies have indeed identified the discrepancy between real-time data collection and retrospective self-report methods [22, 23], emphasising their importance for researchers in capturing less biased, contextually relevant data from participants.
2.1.1.3 Dynamic and Contextual Associations
The longitudinal aspect of experience sampling enables the collection of ecologically valid data for sustained periods of time and in a variety of contexts. In contrast, other assessment methods involving cross-sectional data collection limit possible analyses to differences between participants at specific contexts or instances of time. In capturing within-participant differences across time and contexts, ESM data offers a variety of analysis opportunities. Shiffman et al. characterise four such uses [13], which are explained and justified with additional examples.
Individual differences
Complementary to providing insight into within-person variations, experience sampling data can also beaggregatedacross time to provide detailed insight into individual differences. In such analyses, participants themselves are independent variables, and differences between participants at specific points in time have greater validity, as these differences are aggregated from multiple data points, as opposed to a single cross-sectional sample. Examples of questions that can be answered include whether men and women differ in how they experience a particular event, or how the daily emotional experiences of those with mental disorders differ from a control group.
16 CHAPTER 2. EXPERIENCE SAMPLING
Myin-Germeys et al. used experience sampling to distinguish levels of emotional reactivity between participants with three different psychological disorders [24]. Trull et al. similarly distinguished emotional instability between participants with borderline personality disorder (BPD) and those with depressive disorder [25]. While the emotional instability of the BPD participants in comparison to the depressed participants was known prior to the study, these results validated experience sampling for exposing such differences.
Describing history
While aggregation of ESM data is useful for obtaining accurate between-participant variation, a key benefit of experience sampling is that the resulting longitudinal data exposes within- participant differences. This is particularly important for assessing traits that can fluctuate rapidly over time, allowing researchers to study dynamic processes. Ebner-Priemer and Trull discuss this in their review of ESM in mood disorders, where studies were used to quantify the instability of individuals with BPD over a period of time [26].
Scollon et al. refer to within-participant analyses as idiographic research, as opposed to nomotheticresearch [12], which investigates group variations. In the clinical research domain, repeated, idiographic assessments are necessary to capture the dynamic factors of certain conditions. For example, instability cannot be observed in cross-sectional assessments of other conditions characterised by rapid fluctuations, such as psychosis and bipolar disorder.
Temporal sequences
Insights obtained from within-participant variations in idiographic research are largely inde- pendent of the sequence in which these variations occur. However, Shiffman et al. describe how the temporal sequence of states, in addition to their length and magnitude, can provide a further source of analysis. A number of studies have investigated the contextual factors that surround the occurrence of events in particular domains. For example, Haedt-Matt and Keel present a meta-analysis of studies using ESM to investigate the precursors and consequences of binge eating [27]. A study by Mitchell et al. found significant antecedents and consequences of smoking behaviour through experience sampling, with implications for clinical practice [28].
Contextual factors
Time is just one constituent of the context surrounding participants in their daily experiences. Further sources of context can be acquired in a number of ways - for example, they can be assessed through standard self-report measures, by explicitly asking participants where they are, whom they are with, or what they are doing. Shiffman et al. found significant correlations between smoking behaviour, location and presence of others by capturing these variables through self-
2.1. OVERVIEW 17
report [20]. Additionally, certain contextual information can be captured through external devices. For example, Dunton et al. were able to investigate the contextual factors of physical activity by employing accelerometers [29]; Ilies et al. found associations between affect and cardiovascular functioning in a workplace environment with a wearable heart-rate monitor [30]. In recent years, contextual variables have also been captured by sensors embedded in smartphones, the capabilities and advantages of which have been revered for behavioural and clinical research [7, 31]. Further detail on sensor use in ESM is discussed in Section 3.1.
Treatment monitoring
Shiffman et al. do not explicitly mention treatment monitoring in their applications of experience sampling, instead focusing on its usage in research. However, such an application offers unique advantages in clinical practice. For example, frequent travel to appointments is time-consuming and costly, as well as physically difficult for patients with disabilities. As a potential solution, the remote monitoring of symptoms through ESM could reduce the number of routine face-to-face appointments that are required. This reciprocally benefits clinicians themselves, for whom time is saved through supporting patients’ independence in monitoring their own health.
In one study, Wichers et al. reviewed the potential for ESM to assist in the treatment of patients with clinical depression, with quantified, visualised feedback on symptomatic states allowing for “patients and clinicians alike to understand, modify and track the experiences that are currently subsumed under the ‘black box’ diagnosis of depression”[32, p. 264]. A recent review by van Os et al. further discusses the advantages of smartphone technology to ESM in practice [33]. Examples include the regular assessment of symptoms and emotions to identify optimal doses of medication, and to uncover correlations between co-existing symptoms. ESM can even promote positive mental states, where the simple act of self-monitoring and receiving feedback can act as a form of behavioural intervention [34].
2.1.2 Challenges
Although ESM provides a number of benefits for research and practice, the nature of such an intensive data collection protocol innately has complications, which this section will discuss in more detail. While some of these disadvantages are agnostic of technology used, others have been mitigated in recent years through smartphone-based applications. Indeed, technological advancements that could further alleviate the following challenges are an active area of research.
18 CHAPTER 2. EXPERIENCE SAMPLING
2.1.2.1 Participant burden
The completion of assessments at various times of the day can interrupt the normal activities of participants’ lives. Traditional paper diary studies are particularly burdensome, where in addition to mental disruption caused by unexpected prompts, participants are physically encumbered with these paper diaries, which must be on-hand for the duration of a study. Indeed, physical burden is not limited to paper-based methods; electronically administered ESM studies often require that participants carry cumbersome, bespoke study devices. Furthermore, if an assessment schedule is too intense, or does not fit with participants’ own personal schedules, study attrition (drop-out) is a risk. Participants’ compliance is contingent on a number of external factors, but studies tend to show a significant decrease in both data quantity and quality over as little as seven days [35]. Attrition and non-compliance are major issues that require a careful balance of assessment frequency, length, and timings that will vary between and within groups of participants. Christensen et al. propose that the intensiveness of the sampling schedule and the length of each requested assessment are the greatest contributing factors to participant compliance. As a general guideline, they suggest that each assessment should take no longer than two minutes to maximise completion rates [8].
Due to the onerous nature of experience sampling studies, it has been hypothesised that a “self- selection bias” may occur, whereby those who volunteer to participate in such studies are more likely to be motivated, technologically adept individuals [12]. Experience sampling studies may be particularly burdensome to certain populations, especially when they are administered on smartphones [13]. Elderly populations, children, and those with low socio-economic status, are potentially problematic groups for applying ESM with. Although smartphone-based assessments are designed to lower participant burden, this depends on familiarity with such devices. For example, a study by Harrison et al. showed that the uptake and adherence to a smartphone-based self-help program was inhibited by older adults, who would only use their smartphone for emergencies. Access by those in rural areas with poor network coverage was also limited [36]. Compliance is a complex and unpredictable metric, as shown by Sokolovsky et al. in their study of adolescent smokers [37]. Highly positive and negative moods had deletorious effects on