Development, adherence, and usability study of ESM for post bariatric patients : pilot study

Page |

Master Thesis

Development, adherence,

and usability study of ESM

for post-bariatric patients

Page | i

Master Thesis

Development, adherence, and usability

study of ESM for post-bariatric patients

Pilot study

L.T.J. van der Ham, MSc

[email protected] S1987356

Master Health Sciences

University of Twente

Supervisor: Prof. Dr. M.M.R. Vollenbroek-Hutten Second supervisor: Dr. M. Poel

Ziekenhuisgroep Twente

Supervisor: J.G. Timmerman, MSc Second supervisor: Dr. M.J. van Det

27 November 2018

Picture cover page:

Page | ii

Abstract

Introduction. Currently post-bariatric patients are not always able to sustain the needed behaviour change and consequently may gain weight. This study had the goal to develop a tool which measures postoperative predictive behavioural parameters of weight regain of bariatric patients, to give the post-bariatric professionals and researchers insight in patients’ behavioural patterns. The insights gained with this tool can promote a more tailored support and, via that, sustain behavioural change and with that prevent weight regain.

Method. First, postoperative predictive behavioural parameters of weight regain were obtained via interviews with post-bariatric care professionals. Second, the ESM [Experience Sampling Method] procedure and questions were created based on the selected parameters. This was done with help from literature studies and expert meetings with post-bariatric care professionals and ESM developers. Third, the ESM tool was evaluated by measuring short-term adherence of post-bariatric patients to the tool during a two-week pilot. After which, patient satisfaction of the usability and ESM procedure was appraised via a survey.

Results.The most important parameters were: regularity of food intake, protein intake, social support, and mood. Per parameter, questions were developed and sent on pre-planned days. The ESM procedure consisted of a smartphone app which gives notifications two times a day for one or two questions during two weeks. The questions had a twenty-minutes answering slot. In the pilot, on average 66.8% of the questions were answered and there was an average response time of 4 minutes and 29 seconds. Patients were satisfied with the usability, the use of a smartphone, time to answer the questions and the clarity of questions on mood and nutrition. However, patients valued the app and future use of the app mostly negative. The main reasons mentioned for this were the lack of added value and lack of depth.

Page | iii

Table of Contents

1. Introduction ... 1

2. Theoretical framework ... 3

3. Methods... 5

3.1. PHASE 1: Parameter deduction ... 5

3.2. PHASE 2: ESM Procedure development ... 6

3.3. PHASE 2: Questions development ... 7

3.4. PHASE 3: Patient pilot ... 7

4. Results ... 10

4.1. PHASE 1: Parameters ... 10

4.2. PHASE 2: ESM Procedure ... 11

4.3. PHASE 2: Questions and frequency ... 13

4.4. Prototype ... 14

4.5. PHASE 3: Short-term adherence ... 15

4.6. PHASE 3: Usability & ESM design ... 16

5. Discussion ... 18

6. Conclusions ... 21

7. References ... 22

8. Appendices ... 27

8.1. Appendix I – Interview guide post-bariatric care professionals ... 27

8.2. Appendix II – Patient information ... 30

8.3. Appendix III – Demographic questionnaire ... 34

8.4. Appendix IV – ESM questions and procedure ... 35

8.5. Appendix V – Evaluative survey ... 38

Page | iv

List of figures

Figure 1 Research integrated in the CeHRes Roadmap (27) ... 3

Figure 2 Overview methods ... 5

Figure 3 Grade rankings of SUS scores (46) ... 9

Figure 4 Overview ESM procedure ... 11

Figure 5 Prototype ... 15

List of tables

Table 2 Patients’ demographic characteristics of the pilot and the survey ... 15

Table 3 SUS per statement (range 0-4: completely not agree – completely agree) ... 16

Table 4 Clarity of usability questions (range 1-5: completely not agree – completely agree) ... 17

Abbreviations

CeHRes Center for eHealth Research ESM Experience Sampling Method HEI Healthy Eating Index

RRD Roessingh Research and Development SUS System Usability Scale

QoL Quality of Life

Page | 1

1.

Introduction

Systematic reviews show considerable positive effects of bariatric surgery on achieving weight loss (1-3). Bariatric surgery includes abdominal surgical therapies to achieve significant and sustained weight loss (4). The surgery is meant for people with morbid obesity who failed to lose weight by conservative means (1), such as pharmacotherapy and life-style modification (5).

However, long-term weight loss is not achieved by each patient. Following surgery, patients are expected to commit to a strict lifestyle program. The success of the surgery depends on the commitment of the patient to these lifestyle changes (6). These lifestyle changes are needed to fully benefit from the acquired postoperative weight loss and to avoid weight regain. That means that a change in patients’ behaviour is necessary to generate lasting weight loss (7). However, patients are not always able to maintain this behaviour change and can consequently gain weight. Therefore, patients should be supported in their behaviour change to improve surgery outcomes.

Currently, a postoperative lifestyle program is available at ZiekenhuisGroep Twente [ZGT] obesity centre aiming to support a patient’s lifestyle change after surgery. The program consists of mandatory and optional supportive group meetings, available until five years after surgery (8). The meetings are hosted by a dietician, an obesity nurse, a psychologist, or a physical therapist.

The obesity centre aims to develop a tool as an addition to standard postoperative care to improve treatment outcome. This tool should measure postoperative predictive behavioural parameters of weight regain of bariatric patients and give insight in patients’ behavioural patterns to professionals, so they can see when some patients relapse in unhealthy patterns. Developing such a tool was clinically relevant as these insights can promote a more tailored support and, via that, sustain behavioural change and with that patients’ weight loss.

Mundi et al. (6) mention that eHealth can enhance traditional weight loss programs, for instance by text messages (9), virtual coaching (10), and social media (11). These studies imply that patients are accepting this technology, and even frequently prefer technology over traditional methods. Therefore, for this study eHealth was selected. eHealth is defined by the WHO as “the use of information and communication technologies (ICT) for health” (12).

A subtype of eHealth is mHealth. This is an eHealth application carried out with mobile technology, like smartphones and tablets (13, 14). In 2014, already 24 mobile technology applications for patients relating to bariatric surgery has been identified by Stevens et al. (15). These applications contained information for patients, support forums, advertisements for weight loss clinics, and patient record tools. These patient record tools focused on clinical and organisational aspects, like weight tracking and appointment overviews. For this study a tool is needed to get insight in postoperative predictive behavioural parameters, as the professionals wanted to prevent relapse. However, no similar tool was found by Stevens et al. (15). Therefore, a new tool needed to be developed to make the behavioural patterns measurable.

A method that measures daily behaviour patterns is the experience sampling method [ESM].

Page | 2

someone’s daily life instead of a snapshot. Third, conducting observations or questionnaires usually takes a long time and can create disturbances in a person’s life, which influences the results (16). With ESM, answering a question takes subjects usually less than one minute, which causes only a small disturbance in their day. Finally, since ESM enables measurement in the natural setting, interpersonal and environmental factors can be measured, unlike in laboratory studies (17). With ESM, the target group should answer questions throughout the day, while they are in their natural setting (18). The questions can be prompted at ‘random’ times per day for several days and result in an individual pattern for each patient (19). ESM provides a window in daily behaviour and experience by collecting information about the content and context of daily life (20, 21). There was no known ESM application available to measure predictive behavioural parameters of weight regain for patients after bariatric surgery at the time of the study.

Also, no research had been conducted about the development or feasibility of an ESM module targeted on post-bariatric patients. This means that there was a gap in scientific knowledge about the ESM in post-bariatric care. This study sets the first steps into bridging this gap. This was done by developing and evaluating an ESM application that measures postoperative predictive behavioural parameters of weight regain of post-bariatric patients in a pilot study.

This study had the goal to develop a tool which measures postoperative predictive behavioural parameters of weight regain of bariatric patients, to give the post-bariatric professionals and researchers insight in patients’ behavioural patterns, for which an mHealth ESM tool was selected. To achieve the goal the following research questions and sub questions were answered:

1) Which ESM procedure and questions are best suited to measure postoperative predictive behavioural parameters of weight regain of post-bariatric patients in an mHealth application?

1a) What are the most important postoperative predictive behavioural parameters of weight regain of post-bariatric patients, which can be influenced and measured with an mHealth application, according to professionals involved in bariatric care in ZGT?

1b) What are the variable components of an ESM procedure and which options within these components are preferable to measure postoperative predictive behavioural parameters of weight regain of post-bariatric patients?

1c) What questions and frequencies of those questions are best suited to measure the selected parameters, taking the selected ESM procedure into account?

2) What is short-term adherence of post-bariatric patients to and their opinion of the usability and ESM design of an ESM module that measures postoperative predictive behavioural parameters of weight regain in post-bariatric patients?

Page | 3

2.

Theoretical framework

This study needed a guideline to develop and evaluate an eHealth tool to monitor patients. The CeHRes [Center for eHealth Research] roadmap was used to structure this study (see Figure 1), because of the following arguments. Firstly, because it describes the development of eHealth as a holistic principle by mentioning several steps. It does not merely include one aspect as other eHealth frameworks, e.g. solely include business modelling (22) or evaluation (23). This fits with the study, because the study focuses on development and evaluation. Secondly, with the CeHRes roadmap it is possible to create or adapt a technology useful for only monitoring and not necessarily changing behaviour, as was done in many other frameworks. This corresponds with the aim of the study, because the study aims to monitor and not change behaviour.

Figure 1 Research integrated in the CeHRes Roadmap (27)

The CeHRes Roadmap (24) is an approach for the development of new technologies, the improvement of existing technologies, and the evaluation of technologies. The roadmap consists of five steps. After each step, a formative evaluation takes place to take a step back and optimize the product (25-27). The first step, the contextual inquiry, is about generating insights and understanding of the context (25, 26, 28). In the value specification, the second step, the values of the selected stakeholders should be brought to light (25, 26, 29). Thereafter, (a prototype of) the technology is developed, in the third step, the design step (25). In the fourth step, the operationalisation, the technology is launched (25, 26). Summative evaluation is the last step of the roadmap, which is used to measure what has been accomplished at a certain point in time (30). In this step, the usage of the technology and the effects on healthcare and patients should be studied (25).

In this study, the value specification and design step were carried out. The previous step including the context (step 1) was shortly described in the introduction. Phase 1 of this study corresponds with the value specification step. In this phase the parameters and their importance were derived. Phase 2 matches with the design step, this includes the development of the questions and the ESM procedure. Further steps were not included as this was not the goal of this study.

Page | 4

roadmap, a researcher is in (31). Furthermore, in this study was decided that the prototype was evaluated by end users, to be able to match the technology to the users (32). By using user participation, a technology is created that meets users’ expectations and context (33). When focusing on users it is important to use a mixed method approach by placing a focus on their use behaviour and on their opinion of the app. This is because users are not always aware of or able to express how they use the app (34, 35). Therefore, the following aspects were selected to evaluate: adherence (use behaviour) and content of ESM and usability (opinion of the app).

Adherence should be used to measure the use behaviour after the design, because by measuring adherence, there can be tested if the intended objectives from the value specification has been realised in the design step (30). Adherence is often used as an outcome measure when evaluating eHealth. However, there is no standard adherence definition in eHealth research yet (36). For this study the definition of Donkin et al. (37) was selected: “the degree to which the user followed the program as it is designed”. The definition was combined with the elements of adherence that were mentioned in a recent systematic review to clarify adherence (36): “1) the ability to measure the usage behavior of individuals; 2) an operationalization of intended use; and 3) an empirical, theoretical, or rational justification of the intended use.”.

Page | 5

3.

Methods

The following chapter describes the performed methods for this study in more detail. Figure 2 gives a graphical representation of the methods.

Figure 2 Overview methods

3.1.

PHASE 1: Parameter deduction

The methods of phase 1 include interviews with health care professionals, partly based on literature. This method was selected to get in-depth insight in clinical experience in the ZGT context, without neglecting scientific evidence. In the interviews, post-bariatric predictive behavioural parameters of weight regain were deducted. The intended outcome was an overview of the most important postoperative predictive behavioural parameters of weight regain which can be influenced and measured with an ESM application, according to the professionals.

Participants. Professional who work in the current post-bariatric surgical care of ZGT were asked to participate. These professionals were selected because it was expected that they had insights in the predictors of weight regain after surgery, as they encountered the patients after the surgery. Professionals were included when they had contact with patients in the postoperative lifestyle program and excluded when they only had facilitating or administrative tasks in the postoperative lifestyle program.

Eleven professionals were selected from the information folder of the postoperative lifestyle program by obesity centre intern C. Mels and were approached via email. Five interviews were conducted with seven professionals: one dietitian, one psychologist, one psychological assistant, one physiotherapist, two specialised nurses, and one bariatric nurse. The other approached professionals were not available for an interview.

Page | 6

parameters. In the third part, professionals were asked which parameters mentioned in the interview were the most important according to them. Each professional was asked about parameters belonging to their expertise, e.g. only dietary parameters to a dietician. Nurses were asked about all parameters, as they talk with patients about a range of topics.

Procedure. The face-to-face interviews were alternately conducted by C. Mels and L. van der Ham (researcher). The interviews took about 30 minutes and were held in Dutch. During the interviews, minutes and audio records were made alternately by C. Mels and L. van der Ham. Permission from the professionals was asked to make the records.

Analysis. The minutes were structured based on the factors for weight regain mentioned by Karmali et al. (39). They describe the aetiology of weight recidivism in five groups: hormonal/metabolic, dietary non-compliance, mental health, physical inactivity, and anatomic/surgical. This study focuses solely on measuring patients’ behavioural change, therefore only the patient-related causes were selected: dietary non-compliance, mental health, and physical inactivity. The sub-codes were the parameters mentioned during the interview. After structuring the minutes, the audio records have been listened to, to check if no information was missed.

3.2.

PHASE 2: ESM Procedure development

The methods of procedure development in phase 2 involves a mixed method design, combining scientific literature and expert meetings. These methods were selected to get scientific knowledge, but also to get advice from experienced ESM developers and implementers. The intended outcome was an overview of the components within an ESM procedure and the preferable choices within those aspects fitting the aim of this study.

Participants. Experts with expertise on the implementation and execution of the ESM application were sought in the network of the researcher and the ZGT supervisor (epidemiologist). Three experts were found, contacted via email, and meetings were held with all of them.

Instruments. A semi-structured guide was developed in collaboration with both ZGT supervisors (epidemiologist and surgeon). In this guide, the experts were asked for their expertise about the implementation and execution of an ESM application. A focus was placed on the development and substantiation of their ESM application.

Procedure. Consultations were held with two experts via telephone and with one expert face-to-face. During the meetings, notes were made.

Furthermore, the literature search was conducted; the following search terms were used in the databases ‘Scopus’, ‘Google Scholar’, and ‘PubMed’: "ESM" OR "experience sampling" AND protocol OR guide OR procedure. Moreover, snowball sampling was used with the found literature. The goal was to find variable ESM procedure components and options within those components in scientific literature. Literature was included when it mentioned steps including components to develop an ESM procedure and/or options within those components. Literature was excluded when it did not describe experience sampling, when the literature was not written in Dutch or English, or when it was not available via the University of Twente.

Page | 7

3.3.

PHASE 2: Questions development

The methods of the question development in phase 2 include a mixed method design, combining scientific literature and clinical expertise. These methods were selected to get scientific knowledge and to be able to relate it to the context. The questions were developed for the parameters selected in phase 1, with the procedure developed in phase 2 in mind.

Instruments. A semi-structured guide was developed to guide the expert meetings. In this guide the experts were asked for specific questions and frequency to measure the selected parameters.

Participants. Experts with expertise on ‘frequency of food intake’, ‘protein intake’, ‘perceived social support’, or ‘mood’ (the results from phase 1) were sought in the network of the researcher and the ZGT supervisor (epidemiologist). Six experts were found and contacted via email. Meetings were held with four of them (at least one expert per selected parameter), as not all experts were available.

Procedure. The experts were met before the ESM procedure was developed. The consultations were held face-to-face and during these meetings notes were made. Literature was read on basis of recommendations that were made during the expert meetings.

Furthermore, the literature search was conducted after the ESM procedure was developed. In the databases ‘Scopus’, ‘Google Scholar’, and ‘PubMed’ was searched with the queries: “experience sampling” AND intake OR consumption AND food, “experience sampling” AND protein, “experience sampling” AND “social support”, “experience sampling” AND emotion OR mood. The goal was to find studies that use ESM and measure one of the selected parameters. Literature was included when one of the selected parameters was measured with one or two questions or statements (as was decided upon in the selected procedure), ideally with a theoretical foundation. Literature was also included when it mentioned the frequency for measuring one of the selected parameters. Literature was excluded when it was not in English or Dutch or when it was not available via the University of Twente.

Analysis. For each parameter, the selected questions/statements from the studies have been juxtaposed. The question/statement that fits best with the aim of the study and experts’ recommendations was selected. If no question could be selected (as no question was found), a question was developed based on the recommendations of the experts and the advice given in the post-bariatric care. The final questions/statements were presented to a dietitian and psychological assistant of the post-bariatric surgical care of ZGT and approved for clarity and content.

3.4.

PHASE 3: Patient pilot

The methods of phase 3 involve the patient pilot, for which the prototype was used. This prototype was evaluated by measuring short-term adherence and the opinion of the patients about the ESM design, and about the experienced usability.

Participants. As many post-bariatric patients as possible were asked to participate in this study, while they were visiting the ZGT obesity centre in week 34 to 36 of 2018. Patients who underwent bariatric surgery and were in the post-bariatric program in ZGT were asked to participate. The patients were selected from all the stages of the post-bariatric program.

Inclusion criteria:

- The patient underwent bariatric surgery.

- The patient is in the post-bariatric program at ZGT.

Page | 8

Exclusion criteria:

- The patient has insufficient command of the Dutch (written) language.

- The patient has insufficient intellectual capacity to understand the study and thus the informed consent.

In total 164 patients were asked, 102 were eligible, 18 filled in the demographic questionnaire, 13 used the app, and 9 filled in the evaluative survey.

Instrument. An online survey was used to measure the opinion of the patients about the ESM design and about the experienced usability; the first part of the survey consisted of the system usability scale [SUS] to measure the patients’ experience of the usability of the application (40). The SUS consists of 10 items measured on a 5-point Likert scale, which gave a final score between 0-100 (after multiplication by 2.5). An example of a question is: “I thought the system is easy to use” (40). The SUS was found reliable and valid in English, and the Dutch version had similar internal reliability (41). The SUS was selected because it was a quick and general measure. This fits to evaluate a pilot, because a comprehensive usability evaluation places an unnecessary burden on the patients while the app was still in the development phase. The Dutch translation of the SUS of Jansen-Kosterink et al. (42) as mentioned by Koning (43) was used in this study where the word ‘system’ was replaced with ‘application’, to make the statements less vague.

The second part of the survey consisted questions about the selected ESM procedure design, overall experience, and points for improvement. These questions were asked, because it involved aspects that can be adapted regarding the ESM method and validate the choices.

Procedure. After consultations with the healthcare professional and during a group meeting in the current post-bariatric program, the researcher introduced the aims and contents of the study to the patients. After this introduction the patient information, the informed consent letter and the demographic questionnaire were given to the interested patients (Appendix II – Patient information). The demographic questionnaire consisted of questions about gender, age, education and weeks since surgery (see ‘Appendix III – Demographic questionnaire’). Patients were asked to sign the informed consent form and fill in the questionnaire at home or at the obesity centre.

After informed consent was obtained, an instruction manual, username and password were sent to the patients via email. Patients were reminded twice to install the app. The second time, they were asked for a reason if they did not want to participate. The ESM module started immediately after the patient installed the app. An overview of the questions can be found in ‘Appendix IV – ESM questions and procedure’. Patients used the app in their daily life for two weeks.

Two weeks after the patients installed the application, the online evaluative survey was sent via email (see ‘Appendix V – Evaluative survey’). Patients were reminded to fill in the survey once and if they did not want to participate, they were asked for a reason.

Data access. The answers to the questions and the log data from the smartphone application were encrypted and saved on a data-storage of RRD. The passwords were stored as a salted hash at the RRD server to be sure that they cannot be traced back to the patient. This server was only accessible via a web service over an HTTPS connection. The log data needed for analysis was retrieved from the data server by the researcher and was stored in a private folder on the ZGT server. This private folder on the ZGT server was only accessible to the research team involved in the study.

Page | 9

researchers involved in the study had access to the Qualtrics account. The selected data for analysis was retrieved from Qualtrics by the researcher and was also stored in the private folder on the ZGT.

Analysis. The log data from the application was used to measure adherence. In this study the intended use was based on the core concepts of ESM: 1) patients install the app, 2) patients get a notification, 3) patients answer the questions, and 4) patients answer the questions in real time. These points were operationalised as 1) number of people that installed the app, 2) number of notifications received, 3) number of answered questions, and 4) response time. The ‘number of answered questions’ and ‘response time’ were used to validate choices made for the ESM procedure (see 4.2). This was done by comparing the ‘number of answered questions’ and the ‘response time’ between the selected questions, selected days, and selected times. Also, by showing an overview of the ‘number of answered questions’ and the ‘response time’ from the first to the last question and a response time distribution.

To analyse the experienced usability, the SUS score (from 0-100) was calculated to get a general indication of the user’s subjective rating of the usability (40, 45). The items 1,3,5,7, and 9 score from 0-4 and the items 2,0-4,6,8, and 10 score from 0-4-0. The scores were summed and multiplied by 2.5 to get the overall SUS score (40, 43). In a study of Bangor et al. (46), grade rankings were given to the SUS scores (see Figure 3), to which the score from this study was compared.

To analyse the opinion of the patients about the ESM design, the answers on the qualitative questions of the survey were sorted by: overall experience, points for improvement, and ESM procedure.

Page | 10

4.

Results

4.1.

PHASE 1: Parameters

This chapter describes only the parameters that were found the most important. The most important parameters mentioned by nurses were not described, because the nurses were not able to give a substantiation of why the parameters were the most important. A complete overview of the parameters from the literature and the interviews is listed in ‘Appendix VI – Parameters’, with information from the literature and comments of the professionals summarised per parameter.

Protein intake. The dietician mentioned protein intake as one of the most important parameters. She said that the intake of enough proteins is a good basis for a wholesome diet and makes sure that patients are not hungry. The intake of too little proteins can cause sweet food cravings, a higher intake of fat, and/or hunger. The specialized nurse also mentioned that if patients eat not enough protein, they have a lower feeling of fullness and crave sweet food. She added that patients can feel tired and sick, due to a shortage of proteins. The dietician mentioned that patients’ protein target is based on their ideal weight; patients need one gram of proteins per kilo ideal weight.

Frequency of food intake. The dietician named frequency of food intake as another important parameter. Frequency of food intake describes the number of times a person consumes food per day. She mentioned that patients should eat multiple times throughout a day, to prevent unhealthy food craving. The specialized nurse added that patients sometimes feels unwell if they do not eat enough. The dietician and specialized nurse both mentioned that patients should eat six to ten times a day.

Mood. The psychologist and psychological assistant mentioned that all parameters are important as they are interconnected. Most of the parameters they mentioned relate to the overall feeling of patients. Therefore, patients’ overall feeling was measured with the help of mood. Mood can be defined as “a conscious state of mind or predominant emotion” (47). This parameter is also closely related to depression, which is seen as one of the more important parameters by the psychologist and psychological assistant as well.

Perceived social support. The psychologist and psychological assistant were especially interested in perceived social support, because this parameter was found important in recent research. They said that patients get difficulties if they do not experience social support. The specialized nurse mentioned that a lack of social support can cause the patients to fall back into old habits. Also, the physical therapist said that patients’ success depends on their social environment. Moreover, the dietician mentioned that patients can also be negatively influenced by their peers, when they bring them in all kinds of temptations. Social support can be defined as “advise on, arrange, or provide practical help or emotional social support (e.g. from friends, relatives or colleagues) for performance of the behaviour” (48). In this study, with social support was meant the perceived support a person gets for their lifestyle change.

Frequency of activity. According to the physical therapist1 _{the frequency of activity was the most} important parameter, because it does not matter if patients exercise very long and intensive if they do it only sporadically. She recommends that patients should be active for 30 minutes per day, but it is more important that patients do not sit too much. Frequency of activity can easily and objectively be measured with a smartphone or smartwatch; therefore, patients were not burdened with a physical activity question in this ESM module.

1 _{The physical therapist only mentioned parameters that cause obesity in the first place, because she only guides}

Page | 11

In summary the parameters, ‘protein intake’, ‘frequency of food intake’, ‘perceived social support’, and ‘mood’ were selected for this study.

4.2.

PHASE 2: ESM Procedure

During the consultations with experts that work with an ESM app, two experts mentioned that the app they used was externally developed and that they did not have insight in the ESM development choices. The third expert recommended to take the number of questions per day and the number of questions per moment into account and to variate questions per day.

In the literature, ESM guides of Conner et al. (49), Csikszentmihalyi et al. (50) and Fisher et al. (51) were found and other studies that mention ESM components (18, 20, 52-54). In these guides, the following procedure components were mentioned: platform, sampling protocol, number of questions, duration of sampling period, and cut-off point. The ESM procedure should be tailored to the purpose of the study (49, 50), therefore each component and options within are described below (see Figure 4).

Figure 4 Overview ESM procedure

Page | 12

which a professional in needed. This is not the case at the paper-and-pencil method (20). This method can be done by the researcher itself.

This study uses a computerised ESM by using a smartphone application. First, because as mentioned in the introduction, patients frequently prefer technology over traditional methods. Second, because the computerised version has more advantages as can be seen above.

Sampling protocol. There are three kinds of sampling protocols mentioned in literature (20, 49, 51, 52, 54): interval contingent, signal contingent, and event contingent. With interval contingent the signals come at a regular interval, thus on fixed times (20, 52). This protocol gives the possibility to ask about a time frame or about experiences since the last question (49). With signal contingent, the questions can be answered on response to a signal. This is meant for behaviour that is ongoing and therefore likely to occur when a signal is given (20, 52). Signal contingent also overcomes the influence of retrospective memory bias or cognitive or emotional regulation (18). With event contingent, questions can be answered directly following a certain event, ensuring a limited memory bias. This protocol is useful for behaviour that is less common throughout a day. A study that measured the most optimal ESM protocol found that response is usually highest when using interval contingent (49).

In this study, the application sent questions to the patient with an interval contingent protocol, because the dietary questions ask about the whole day, and thus include a time frame. Hence, the questions (developed later in phase 2) were set at a certain day and time. Patients got notifications on their phone, in this way a signal contingent protocol was used as well.

Number of questions. In scientific literature, there is variation in the advice about the number of questions. This varies between studies, from 7 to 270 questions per person (49). The number of questions depends on the number of signals (51); 1 signal per day can have questions of 5-10 minutes and 5 signals per day can have questions of maximum 3 minutes. The observations should be often enough to capture fluctuations, but too often burdens the patients (49). The more questions that are asked, the fewer patients are willing to participate (50). However, when this is set too low there is the possibility of not having enough data. When items are concrete, a single well-chosen item should be sufficient to measure the construct (51). No more than 30 items per moment and 8 signals is feasible in clinical practice (18, 53). Delespaul (55) advices against more than six question moments per day when using interval- and signal-contingent for a longer period of time. Question moments once or twice a day generally gives the highest response (49). In one study with pre-bariatric patients, eight questions per week were used (6). Another study, which sent text messages to obese patients, found that more than two text messages per day is too many for this target group (56).

The expert also mentioned the number of questions as an ESM component. She mentions that a low number of questions was needed to not overburden the patient, and that false answers and dropouts occur with a lot of questions.

In this study, questions were asked twice a day, as twice a day was seen as the highest possibility for this target group without burdening the patients too much (56) and as it usually gives the highest response (49). The number of questions was set to one or two per moment with a preference for one, to decrease the chance of overburdening patients, and thus decrease false answers and dropouts.

Page | 13

In this study, the questions were asked for two weeks. The sampling period had to be short, due to time limitations. However, patients needed to be able to evaluate each question. If the sampling period only lasted one week, there was a high chance that patients did not see some of the questions.

Cut-off point. The idea behind ESM is a momentary assessment, therefore the answers have to be given within a certain time; a cut-off point is needed to get ‘real-time’ insights (20, 51). In a book chapter of Csikszentmihalyi et al. (50) studies are mentioned in which 50% and 64% of the patients responded immediately. In one of those studies 87% responded within ten minutes and in the other 90% within eighteen minutes. Csikszentmihalyi et al. (50) also mention that several studies have reported a similar delay of response and most studies use a cut-off point of 20 minutes.

In this study, the cut-off point was set on twenty minutes, as it is close to a momentary assessment while usually most patients have responded.

4.3.

PHASE 2: Questions and frequency

This chapter describes the selected question/statements and frequency for the parameters selected in 4.1 based on the procedure selected in 4.2 (see Table 1 or Appendix IV – ESM questions and procedure). There were two time-possibilities set for each question, to get insight in the responses per time. Each question was asked one time the first week and the other time in the second week. Patients were randomly assigned to these times. An answer option ‘I do not know’ was added to every question for patients do not have the knowledge to answer the questions.

Table 1 ESM questions/statements and answer options

Question Answer options

How often did you eat today? 1-5 times / 6-10 times / more than 10 times / I do not know Have you eaten enough protein today? Yes / No / I do not know

I feel supported in complying with dietary advice. Strongly agree / Agree / Neutral / Disagree / Strongly disagree I feel supported in complying with exercise advice. Strongly agree / Agree / Neutral / Disagree / Strongly disagree How do you feel now? ☺ /  /  / I do not know

Frequency of food intake. An expert (master student nutrition and health) recommended to ask for the number of eating moments three times a day to measure frequency of food intake while avoiding memory bias. Moreover, the dietician mentioned that patients have different routines in weekdays and weekends.

In the literature no method was found to measure food intake frequency in one question (as was decided upon in the procedure). Frequency of food intake is usually measured with a food frequency questionnaire, recall, or food diary. These are usually filled out three days per week to get a valid measure (two weekdays and one weekend day) (57-59).

To measure frequency of food intake in this study, a tailored question was developed with the advice given in the post-bariatric program care kept in mind. This advice is: to eat six to ten times a day. The frequency of three days per week with two weekdays and one weekend day was selected, because this was advised by the literature and the dietician. The question was asked at 21.00 or 22.00, as a retrospective view on the day was required to answer this question.

Page | 14

In the literature no method was found to measure protein intake in one question. Protein intake is also usually measured with a food frequency questionnaire, recall, or food diary. These are usually filled in three days per week to get a valid measure (two weekdays and one weekend day) (57-59).

A tailored question was made for to measure protein intake based on the advice in the post-bariatric program. The advice for level of protein intake differs per person; therefore, the patients were only asked if they meet the required amount (patients get information about their needed personal protein intake). For this a measurement of three days per week with two weekdays and one weekend day was selected as well, because this was advised by the literature and the dietician. The question was asked at 21.00 or 22.00, as a retrospective view on the day is required to answer this question.

Perceived social support. The experts (psychologist and psychological assistant) recommended measuring social support with the ‘Multidimensional scale of perceived social support’. Moreover, they mentioned a research of one of their colleagues measuring social support among bariatric patients, with two questions (60). According them, asking for social support one time a week would be enough for a long-term overview.

In scientific literature several articles describe perceived social support. However, most measure social support with a questionnaire of multiple questions (61-69). Pacella et al. (70) and Hatzenbuehler et al. (71) describe a perceived social support measure with one or two questions. Many studies mention measuring social support daily (65), or even multiple times per day (61, 72, 73).

For social support the questions were based on the research of Ter Braak et al. (60), because these were developed for Dutch post-bariatric patients. The question asks about support for compliance with nutritional and exercise advice on a 5-point scale. The questions were asked one time a week, following the advice of the experts. The questions were asked at 10.00 or 18.00, to look for a difference between expected work time and leisure time.

Mood. The experts (psychologist and psychological assistant) recommended measuring mood with the ‘Positive and Negative Affect Scale’ or the ‘Nederlandse vragenlijst voor eetgedrag’ and if there was a focus on depression, the ‘Beck Depression Inventory’. They recommended to measure mood daily, because it can have a lot of variation within a short time.

During the literature search, a visual method was sought to measure mood (e.g. with emoticons) with the notion that visual scales are better understood by patients (54). Several studies were conducted in which mood was measured using ESM. However, no study measures mood with one valid question. The ‘facial image scale’ was the most commonly described scale when searching for smiley scales. This scale normally uses five smileys, but is adapted by Lee et al. (74) to three smileys, as people experienced it as hard to differentiate between very sad and sad, and happy and very happy. These three smileys can also be used as a measure to look for depression, as it had strong correlations with the ‘Geriatric Depression Scale’ and depression in the DSM IV (74).

To measure the parameter mood, the question of Lee (74) was selected. Mood was measured daily, due to the variation within the parameter. The question was asked at 12.00 or 19.00, to look for a difference between expected work time and leisure time.

4.4.

Prototype

Page | 15

Figure 5 Prototype

4.5.

PHASE 3: Short-term adherence

Sample. In total 164 patients were approached in three weeks, of which 62 patients were excluded due to the incompatibility with an iPhone (n=54), due to a lack of a smartphone or smartphone knowledge (n=4), and due to insufficient command of the Dutch (written) language (n=4). The other 102 patients belonged to the study population, of which, 6 (5.9%) patients did not want to participate due to time limitations, 2 patients (2.0%) did not participate due to other reasons, and 12 patients (11.8%) gave no reason. This leaves 82 patients (80.4%) with a positive intention towards participating. Those patients received the informed consent letter and demographic questionnaire. In total, 18 patients (of 102 ,17.6%) completed these forms (see Table 2 for an overview of the patients’ demographic characteristics). The reasons for not completing the forms were not stated by non-participants.

Table 2 Patients’ demographic characteristics of the pilot and the survey

Demographic characteristics Patients who completed the forms (n=18)

Patients who used the application (n=13)

Patients who completed the evaluative survey (n=9)

Female, n (%) 17 (94.4%) 13 (100%) 9 (100%) Age, years 42 [34.75-50.75] 43 [34.5-50] 46 [38-51.5] Highest completed educational level

- Primary education, n (%)

- Lower Vocational Education (VMBO), n (%) - Intermediate vocational education (MBO), n (%) - Higher general continued education (HAVO) or

pre-university secondary education (VWO), n (%) - Higher vocational education (HBO), n (%) - University (WO), n (%)

1 (5.6%) 2 (16.7%) 11 (61.1%) - 3 (16.7%) - - 2 (15.3%) 8 (61.5%) - 3 (23.1%) - - 1 (11.1%) 6 (66.7%) - 2 (22.2%) - Weeks since bariatric surgery, weeks 9.5 [6.75-59.5] 6.5 [8-49] 8 [6.5-27]

Page | 16

Validation of procedure. No visual difference in response rate was seen between the different questions; the percentages were around 67%. A difference was seen in response time; the question about protein was answered faster than the other questions, respectively 3 minutes and 50 seconds (noted as 3:50) compared to 4:32. A difference in response rate was seen in day and time; on Tuesday patients answered more often (81.3%), followed by Monday (70.8%) and Saturday (69.6%), on the other days the percentages were around 62%. The response time was the fastest on Sunday (3:46) and the slowest on Monday (5:37), the other response times were around 4:30. The questions received at 10.00 am (46%) and 12.00 pm (39%) were answered the most, and the least at 6.00 pm (20%). The percentages of the other times were around 30%. The questions were answered the quickest at 10.00 pm (3:33), and the least at 7.00 pm and 9.00 pm (4:58 and 4:53). The other answering times were around 4:25. Most patients answered the questions quickly; of the total given answers, 50% was given within 2 minutes, 65% within 4 minutes, and 83% within 10 minutes. The sampling period lasted 14 days, the response rate decreased during the 14 days and the response time remain variable the whole time. Relating to the response time distribution, 49.8% of the answers were given within 2 minutes, 67.7% within 5 minutes, and 87.2% within 10 minutes.

4.6.

PHASE 3: Usability & ESM design

Usability. The average SUS score was 77.8 (σ=12.9). This falls within the acceptable range and between the good and the excellent score determined by Bangor et al. (46). Most statements were scored positively (see Table 3). Only the statements ‘I think that I would like to use this app frequently’ and ‘I found the various functions in this app were well integrated’ had remarkably low scores, a respective average of 1.4 and 2.0 (range 0-4: completely not agree - completely agree).

Table 3 SUS per statement (range 0-4: completely not agree – completely agree)

Statement Mean (SD)

I think that I would like to use this app frequently 1.4 (1.0) I found the app unnecessarily complex 3.7 (0.5) I thought the app is easy to use 3.8 (0.4) I think that I would need the support of a technical person to be able to use this app 3.7 (0.5) I found the various functions in this app were well integrated 2.0 (1.3) I thought there is too much inconsistency in this app 3.1 (1.1) I would imagine that most people would learn to use this app very quickly 3.6 (0.7) I found the app very cumbersome to use 3.6 (0.7) I felt very confident using the app 2.6 (1.0) I needed to learn a lot of things before I could get going with this app 3.8 (0.4)

Page | 17

value (n=2), everything (n=1), nothing (n=1), little (n=1), lack of information (n=1), no in-depth picture with these questions (n=1), the possibility to go on autopilot due to the repetition of questions, and no stimulation or motivation to do it right (n=1). One patient dropped out due to the lack of variety of the questions. Patients suggested to add: a higher variety of questions (n=2), other questions (n=1), more questions with more depth (n=1), more depth and precision in the questions (n=2), more functions (n=1), a possibility to add questions yourself (n=1), possibility to unsubscribe for questions (n=1), and a dashboard with results (n=1). One patient mentioned that she did not understand the goal of the app, as she was not able to do anything.

Most patients did not want to use the app as part of their treatment (n=6), one patient was neutral, and two patients did want to use it as part of treatment (μ=2.2, σ=1.2, range 0-4). Most patients did not recommend this app to other bariatric patients (n=6), two were neutral, and one would recommend it (μ=2.2, σ=1.0).

The questions were generally seen as clear, with an average score of 3.9 (range 1-5: unclear - clear). The social support questions scored remarkably low on clarity with 3.1 (see Table 4). According to three patients, the questions about support could use more context by adding from whom they felt supported. On other questions was commented on aspects besides clarity; a lack of depth within the nutrition and mood questions (n=1), too low frequency of the ‘frequency of food intake’ question (n=1), and according one patient was the mood question only a snapshot as mood varies throughout the day. Also, two patients mentioned missing a question about fibre intake.

Table 4 Clarity of usability questions (range 1-5: completely not agree – completely agree)

Clarity of questions /statements Mean (SD)

How do you feel now? 4.7 (0.5) How often did you eat today? 4.3 (0.7) Have you eaten enough protein today? 4.3 (0.7) I feel supported in complying with dietary advice 3.1 (1.5) I feel supported in complying with exercise advice 3.1 (1.5)

Page | 18

5.

Discussion

The aim of this study was to develop a tool which measures postoperative predictive behavioural parameters of weight regain of post-bariatric patients to give the post-bariatric professionals and researchers insight in patients’ behavioural patterns. In this study, a tool was developed to query the predictive parameters of weight regain. Therefore, it could be said that the goal was achieved. However, the patients were negative about wanting to use the app, which was mainly due to the lack of functions of the app; in the current app patients could solely answer questions and got nothing back. The app can only be an addition to the current care if it is appreciated by the users. Therefore, it is recommended to adjust the aim and give direct support to the patients, instead of only gaining insight and giving support via the professionals. In the development of a next version of the tool, it is recommended to use a behaviour change model next to the CeHRes roadmap, because the direct support is about supporting or maintaining behaviour change. Even though the CeHRes roadmap can be used for behavioural change, it does not give much guidance for it. Several behaviour change models are available, like the Fogg Behaviour Model (75), Transtheoretical model (76), and Theory of Planned Behaviour (77). A model should be selected depending to the final goal aim of the tool.

Furthermore, adjustments need to be made regarding the content: the questions and the protocol. The social support questions should be changed, because they were seen as unclear by the patients. According to them, it should be added to the questions by whom patients felt supported. Furthermore, they mentioned that questions about fibre intake could be added to the list of questions.

With regard to the protocol, adaptations could be made as well. The selected interval contingent protocol had the same questions repeated at the same time. In the data can be seen that patients answered the questions faster at the end of the pilot. The repetition of the questions and times could cause familiarisation (78, 79) creating a bias as patients already expect a certain question. Indeed, one patient also mentioned that the repetition caused an autopilot response. This point should be considered relating the quality of the study, as it can induce false answers and dropout. A signal contingent protocol could be used as a protocol for mood and social support to overcome this problem for those parameters. Another point is that the component ‘number of questions’ was set on two questions moments per day, with a maximum of two questions to not overburden the patient. This was based on research on obese and bariatric patients (6, 56). However, this study contradicts these findings; most of the patients said that the number of questions and questions per moment were too little. Patients would have liked more and a higher variety of questions. Furthermore, the pilot period was set on two weeks, most patients mentioned that they did not want to use the app after these weeks, due to a lack of depth and the repetition of the same questions. Additionally, in the data a decrease was seen in the response trendline. This could be improved by the previously mentioned implementation of direct support and adding more questions.

Page | 19

themselves assumed this would increase response, due to their authority and connection to the patients. Moreover, during the recruiting process it became clear that many patients were hesitant to participate because they had to answer within 20 minutes. Many patients said they were not able to do so, because of work. Therefore, it should be better explained to them that it is fine to miss questions. Moreover, to increase the completion of informed consent forms, it is an option to give the information before the patients come to the centre and ask them to bring it with them.

It would also be recommended to increase the adherence, because adherence is closely related to an intervention’s effectiveness, as suboptimal use of a tool can undermine its effects (80). A method to increase the adherence is to ask more questions on the days and times on which a higher response was found, because a difference in adherence was found between days and times. Another method to increase total adherence, is to focus on persuasiveness of the tool, as this is an important factor of adherence to the tool (81). An example of a framework that can be used for developing and evaluating persuasive systems is the Persuasive System Design model (82). This model e.g. includes personal insight, which was also mentioned by all the post-bariatric care professionals as an important aspect; they mention that patients need insights themselves to be successful, because they are the ones who should do it. As mentioned by a patient and an expert, it is preferable to give patients feedback, e.g. an overview of the answered questions.

Methodical consideration. During the development phase, a mixed method design combining qualitative information from practice and scientific literature has been used. Currently patients were only able to comment after the development. A point of improvement for a next version of the tool is to engage patients in the development process. This is also a part of the evidence-based practice and the CeHRes roadmap (83). By getting direct insight in their views, less often adjustments need to be made, as in this manner issues can be found and addressed immediately.

The validity of the selected parameters and questions could be questioned, because there seems to be no overlap between literature and practice for selecting the parameters; the selected parameters for this pilot all came from practice and were not mentioned in the selected literature. This is caused by the fact that the professionals did not agree with most of the parameters from literature or mentioned that it was important but was dealt with pre-surgery. Furthermore, even though the questions were developed with the help of literature and experts, there was no insight in the validity and reliability of the questions. To get insight in the validity of the questions, a long term study is needed where the results are compared with one time clinical measures of the selected parameters (84).

It is expected that the study population was not a representation of the target population. It is likely that only patients who were interested in research participated, known as selection bias (85). The self-selection and attrition bias is often created during an ESM research, as only certain patients apply to participate and a certain group usually answer to the ESM questions (54). It should be noted that the motivation of patients to answer the ESM questions correlate with their motivation to change (18). It is expected that currently only patients with a motivation to change were interested in participating.

Page | 20

Further research.

The next step after this study would be to improve the ESM tool, according to the CeHRes roadmap. This pilot study intended to set the first steps into bridging the gap in scientific knowledge about the ESM in post-bariatric care. It was a broad study and has touched upon many different parts of the development of ESM. Given the limited time, this also means that not all parts were researched as thoroughly as could be. This research should only be seen as a foundation for future versions, as it was developed by one person. Usually an ESM module is developed by a team with each their own expertise, according to two experts and Csikszentmihalyi et al. (50). It would be recommended to do a more in-depth research for each step with a team of experts. Keeping in mind the lessons learned from this study about adapting the aim to behavioural change, improving the ESM questions and the procedure, increasing the participation rate and increasing the adherence.

Page | 21

6.

Conclusions

This study aimed to develop a tool which measures postoperative predictive behavioural parameters of weight regain of bariatric patients, to give the post-bariatric professionals and researchers insight in patients’ behavioural patterns. To reach this goal, two main questions were answered, which were shortly described in this conclusion.

1) Which ESM procedure and questions are best suited to measure postoperative predictive behavioural parameters of weight regain of post-bariatric patients in an mHealth application?

For the ESM procedure, the following components were found: platform, sampling protocol, number of questions, duration of sampling period, and cut-off point. Within those components was chosen for respectively: a smartphone platform, interval contingent protocol, 28 question moments and 1-2 questions per moment, 2-week period, and a cut-off point of 20 minutes.

The most important parameters to be measured with ESM were: regularity of food intake, protein intake, social support, and mood. Per parameter, questions or statements were developed, respectively: ‘How often did you eat today?’ (asked 3 times a week), ‘Have you eaten enough protein today?’ (asked 3 times a week), ‘I feel supported in complying with dietary advice’ and ‘I feel supported in complying with exercise advice’ (both once a week), and ‘How do you feel now?’ (asked daily).

2) What is short-term adherence of post-bariatric patients to and their opinion of the usability and ESM design of an ESM module that measures postoperative predictive behavioural parameters of weight regain in post-bariatric patients?

In total 164 patients were approached for the pilot of the app, installed 13 patients the application and 12 patients got all notifications. On average 66.8% of the questions were answered and there was an average response time of 4 minutes and 29 seconds.

According to patients the prototype was usable, and most questions were seen as clear (except for the social support questions). However, there are still some points for improvement, as patients valued the app and future use of the app mostly negative. The main reasons mentioned for this were the lack of added value and lack of depth.

Page | 22

7.

References

1. Buchwald H, Avidor Y, Braunwald E, et al. Bariatric surgery: A systematic review and meta-analysis. JAMA. 2004;292(14):1724-37.

2. Chang S-H, Stoll CRT, Song J, Varela JE, Eagon CJ, Colditz GA. Bariatric surgery: an updated systematic review and meta-analysis, 2003–2012. JAMA surgery. 2014;149(3):275-87.

3. O'Brien PE, MacDonald L, Anderson M, Brennan L, Brown WA. Long-Term Outcomes After Bariatric Surgery: Fifteen-Year Follow-Up of Adjustable Gastric Banding and a Systematic Review of the Bariatric Surgical Literature.[Article]. Annals of Surgery. 2013;257(1):87-94.

4. Suijkerbuijk AWM, van Gils PF, de Wit GA, Feenstra TL. Kosteneffectiviteit van bariatrische chirurgie: Een review van economische evaluaties. Rijksinstituut voor Volksgezondheid en Milieu, 2012.

5. Gadde KM, Martin CK, Berthoud H-R, Heymsfield SB. Obesity: Pathophysiology and Management. Journal of the American College of Cardiology. 2018;71(1):69-84.

6. Mundi MS, Lorentz PA, Grothe K, Kellogg TA, Collazo-Clavell ML. Feasibility of Smartphone-Based Education Modules and Ecological Momentary Assessment/Intervention in Pre-bariatric Surgery Patients. Obesity Surgery. 2015;25(10):1875-81.

7. Bauchowitz AU, Gonder-Frederick LA, Olbrisch M-E, Azarbad L, Ryee M-Y, Woodson M, et al. Psychosocial Evaluation of Bariatric Surgery Candidates: A Survey of Present Practices. Psychosomatic Medicine. 2005;67:825-32.

8. ZGT. Obesitas behandeling in een gezond gewicht in 8 stappen: ZGT; n.d. [Available from:

https://www.zgt.nl/patienten-en-bezoekers/onze-specialismen/interne-geneeskunde/zgt-obesitascentrum/behandeling/.

9. Cole-Lewis H, Kershaw T. Text Messaging as a Tool for Behavior Change in Disease Prevention and Management. Epidemiologic Reviews. 2010;32(1):56-69.

10. Watson A, Bickmore T, Cange A, Kulshreshtha A, Kvedar J. An Internet-Based Virtual Coach to Promote Physical Activity Adherence in Overweight Adults: Randomized Controlled Trial. Journal of Medical Internet Research. 2012;14(1):e1.

11. Napolitano MA, Hayes S, Bennett GG, Ives AK, Foster GD. Using facebook and text messaging to deliver a weight loss program to college students. Obesity. 2013;21(1):25-31.

12. WHO. eHealth: WHO; n.d. [Available from: http://www.who.int/ehealth/about/en/.

13. van Dyk L. A Review of Telehealth Service Implementation Frameworks. International Journal of Environmental Research and Public Health. 2014;11(2):1279-98.

14. IJntema M. De beste studiecoach vind je in de appstore: Een onderzoek naar bestaande apps die ondersteunen bij uitstelgedrag in het hoger onderwijs. . Universiteit Twente, 2016.

15. Stevens DJ, Jackson JA, Howes N, Morgan J. Obesity Surgery Smartphone Apps: a Review. Obesity Surgery. 2014;24(1):32-6.

16. Larson R, Csikszentmihalyi M. The Experience Sampling Method. In: Csikszentmihalyi M, editor. Flow and the Foundations of Positive Psychology. Dordrecht: Springer Science+Business Media; 2014. p. 21-34.

17. Wilhelm Frank H, Roth Walton T. Taking the laboratory to the skies: Ambulatory assessment

of self‐report, autonomic, and respiratory responses in flying phobia. Psychophysiology. 2003;35(5):596-606.

18. Os J, Verhagen S, Marsman A, Peeters F, Bak M, Marcelis M, et al. The experience sampling

method as an mHealth tool to support self‐monitoring, self‐insight, and personalized health care in

clinical practice. Depression and Anxiety. 2017;34(6):481-93.

19. KU Leuven. ESM: Experience Sampling Method: KU Leuven; n.d. [Available from:

https://gbiomed.kuleuven.be/english/research/50000666/50000673/cpp/ccp-dutch/onderzoek/experience-sampling-method.

Page | 23

21. Hektner J, Schmidt J, Csikszentmihalyi M. Experience Sampling Method: Measuring the Quality of Everyday Life. California: Sage Publications; 2006.

22. Oderanti F, Oluseyi, Li F. Commercialization of eHealth innovations in the market of the UK healthcare sector: A framework for a sustainable business model. Psychology & Marketing. 2018;35(2):120-37.

23. Dansky KH, Thompson D, Sanner T. A framework for evaluating eHealth research. Evaluation and Program Planning. 2006;29(4):397-404.

24. van Gemert-Pijnen JEWC, Nijland N, Appelman B. eHealth-onderzoek in beeld. Betere eHealth technologieën door slimmer ontwerpen. Enschede: Center for eHealth Research (CeHRes), Universiteit Twente; 2011.

25. University of Twente. CEHRES ROADMAP: University of Twente,; n.d. [Available from:

https://www.utwente.nl/en/bms/ehealth/cehres/.

26. Center for eHealth Research and Disease Management. CeHRes Roadmap 2011 [updated 13-01-2012. Available from: http://www.ehealthresearchcenter.org/wiki/index.php/Main_Page.

27. Center for eHealth Research and Disease Management. Formative Evaluation 2011 [updated

3-05-2013. Available from:

http://www.ehealthresearchcenter.org/wiki/index.php/Formative_Evaluation.

28. van Limburg M, van Gemert-Pijnen EWCJ, Nijland N, Ossebaard CH, Hendrix MGR, Seydel RE. Why Business Modeling is Crucial in the Development of eHealth Technologies. J Med Internet Res. 2011;13(4):e124.

29. Center for eHealth Research and Disease Management. Value Specification 2011 [updated 17-01-2012. Available from: http://www.ehealthresearchcenter.org/wiki/index.php/Value_Specification. 30. Center for eHealth Research and Disease Management. Summative Evaluation 2011 [updated

3-05-2013. Available from:

http://www.ehealthresearchcenter.org/wiki/index.php/Summative_Evaluation.

31. Lau F, Kuziemsky C. Handbook of eHealth Evaluation: An Evidence-based Approach. Victoria, British Columbia, Canada: University of Victoria; 2016.

32. van Gemert-Pijnen JEWC, Peters O, Ossebaard H. Improving eHealth. . The Hague (the Netherlands): Eleven International Publishing; 2013.

33. van Gemert-Pijnen L, Karreman J, Vonderhorst S, Verhoeven F, Wentzel J. Participatory Development via User-Involvement-A Case Study about the development of a Web-based Patient-communication system about Methicillin-resistant Staphylococcus aureus. Electronic journal of health informatics. 2011;6(4):28.

34. Nijland N, Verhoeven F. Human-Centered Design in eHealth. In: Gemert-Pijnen JEWCv, Peters O, Ossebaard HC, editors. Improving eHealth. The Hague: Eleven International Publishing; 2013. 35. Jaspers MWM. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics. 2009;78(5):340-53.

36. Sieverink F, Kelders SM, van Gemert-Pijnen JEWC. Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence. Journal of Medical Internet Research. 2017;19(12):e402.

37. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies. Journal of Medical Internet Research. 2011;13(3):e52.

38. International Organization for Standardization. Ergonomics of human-system interaction —

Part 11: Usability: Definitions and concepts 2018 [Available from:

https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en.