Case study: Pilot trial interviews and questionnaires

11.1 Purpose

The pilot trial of the telehealth system was conducted and evaluated to determine whether the design of the system could adequately support post-operative outpatient consultations for the Departments of Orthopaedics and Maxillo-facial surgery at the Royal Children’s Hospital in Melbourne, Australia. This class of outpatient

consultation was chosen because it represents a significant part of the hospital’s work and because the burden of distance falls heavily on the patients and families during this phase of the treatment. These two departments were chosen by the hospital’s Chief of Surgery as having suitable clinical relevance for this trial.

In this study I present the data gathered immediately after each telehealth session and after each block of sessions. The data from these interviews and questionnaires represent what the participants said about their experiences and feelings after the telehealth sessions.

The purpose of this study was to analyse the responses that the participants gave during exit interviews which were conducted with the patients and family after each

consultation and with the clinicians after each block of consultations. This analysis addressed three broad issues:

• Whether delivery of telehealth consultations with this system would be acceptable to the patients and their families

• Whether the mode of clinical contact would be acceptable to the surgeons

• Whether the support provided for managing the patients and presenting clinical examination and findings to the surgeon would be acceptable to the staff performing the role of assistant

This study is a human-centred evaluation of the pilot trial of a broadband telehealth system placed in a tertiary hospital. This evaluation was conducted from the points of view of the participants in the trial and it demonstrates an approach to evaluating such pilot trials by tailoring the exit interviews for each class of participants to best draw on their context, history and knowledge. Each participant has a different point of view of the success of a particular clinical consultation. By drawing on all three points of view we are able, over the period of the study, to evaluate the telehealth system as a whole.

11.2 Method

Pilot trial and data gathering

Two meeting rooms on the eighth floor of the hospital were used for the trial, one as the surgeon’s room and one as the patient’s room. The patients reported to a waiting room on the same floor. At the scheduled time a member of the research team fetched the patient and family from the waiting room, checked that the consent process was complete and took the patient to the telehealth room. The assistant ushered the patient and family in and directed them to their seats. The surgeon, already in place in the surgeon’s room, greeted them and commenced the telehealth phase of the consultation. The researcher remained with the patient and family for the entire time. After the

telehealth phase of the consultation, the surgeon stepped out of his office, walked to the patient’s room and conducted the face-to-face phase.

At the end of the whole consultation the researcher took the patient and family to a separate room and conducted an exit interview with them. This exit interview, which was audio-recorded and later transcribed, consisted of six Likert-Scale items and a number of semi-structured questions as shown in Appendix A. While this exit interview was taking place a second researcher got the surgeon and assistant to complete a brief questionnaire about their impressions of that particular consultation. At the end of each clinic (a half-day block of patients) the researchers conducted an exit interview with the surgeon and separately with the assistant. These interviews also consisted of a number of Likert-Scale items followed by semi-structured questions (Appendix A), which were audio-recorded and later transcribed.

Pattern of the clinics

The trial consisted of three half-day clinics per week for four weeks. Due to prior commitments on the part of the surgeons there was not an exact match of clinics week by week. Two surgeons, however, each attended four clinics and another attended three. Two other surgeons attended a single clinic each. The pattern of the clinics is shown in Table 11.1

Week Clinics Numbers

of patients 1 Orthopaedics (1,2) Limb reconstruction (2) Craniofacial plastics (3) 4 3 2 2 Orthopaedics (1) Hand plastics (4) Craniofacial plastics (3) 4 3 4 3 Orthopaedics (1) Limb reconstruction (2) Orthopaedics (5) 6 3 4 4 Orthopaedics (1) Limb reconstruction (2) Craniofacial plastics (3) 4 2 5 Total 44

Table 11.1. Pattern of clinics during the study, showing the surgeons (coded 1-5) who conducted the clinics.

The three main surgeons, therefore, had an opportunity to refine their use of the telehealth system during the four-week trial.

Question structure

Each group of participants was asked to respond to items or questions which their situation or prior experience qualified them to answer. For example, the patients and families were asked about their personal response to the just-concluded consultation and the surgeons were asked about their professional response. Three broad types of items or questions were presented:

• Items about personal response, such as the statement “I felt that the [subsequent] face-to-face appointment was necessary”

• Items concerning what they were able to do, such as the statements “I could get the doctor to understand and respond to my concerns during the telehealth appointment” and “I was able to direct the assistant”

• Hypothetical items, drawing on both the participant’s experience and expectations, such as the question “Would you have been satisfied to let the patient leave after the telehealth appointment?”

All Likert-Scale items were presented as first-person (“I”) statements with a choice of five response categories: Strongly agree, Agree, Neither agree nor disagree, Disagree and Strongly Disagree. The response options were listed vertically, with “Strongly agree” at the top.

Where possible, the statements were phrased so that a positive sentiment matched the category “Strongly agree”. This might appear to contradict accepted wisdom, in which some questions are phrased in the negative to prevent a “donkey vote” of mechanical ticks to a large number of items by relatively uninvolved respondents. Our situation, however, was different. We had only six “tick the box” items. We had keen

participants. For example, one parent commented that the hospital had been so good to her and her family over the course of treating her child that she was pleased to be able to give something back by taking part in this trial. In the first week of the trial we did have one item on the families’ questionnaire that required parsing a double negative to interpret. We found that the families were so occupied by the reality of the day spent at the hospital that they did not have the mental energy to interpret the question. For the remaining three weeks we re-printed that page of the questionnaire with the double negative removed.

Data analysis

The Likert-Scale items contained five levels of response with “Neither Agree nor Disagree” at the centre. From our observations of the patients/families completing the questionnaires we often saw cases where the sentiment was clear but there were differing thoughts on whether the response should be, for example, Agree or Strongly Agree. Since we have no separate measure of the interpretation of this level of sentiment between differing families, nor between differing surgeons or assistants, I have taken the approach, described by Gardner and Martin, and clumped the responses into three categories – Agree; Neither agree nor disagree; and Disagree (Gardner and Martin 2007) for all three groups of respondents. Some items in the semi-structured interview led to a Yes/Neutral/No interpretation. Note that some data points are missing. Some families did not complete the exit interview, often for simple reasons such as that their parking ticket was about to expire, or they simply did not answer a question in the semi-structured part of the interview. One family did not complete the exit interview.

11.3 Results

11.3.1

Demographic data

Table 11.2 shows the distribution of the patients’ home locations in terms of distance from the hospital. It also shows the way those families responded to the hypothetical question as to whether or not they would have used such a telehealth service for this consultation if it had been available locally.

Patients’ home location Yes No

Distant (4hr travel or more) 4 0

Country Victoria (1.5hr to 3.5 hr travel) 5 0

Regional to Melbourne (1hr to 1.5hr travel) 5 1

Outer Melbourne suburbs 7 2

Melbourne suburbs 14 4

Total responses 83% (35) 17% (7)

Table 11.2: Patients’/families’ home locations and responses to whether they would have used this system if it had been available locally

11.3.2

Patients and families

Likert-Scale questions

Table 11.3 summarises the responses from the patients and families to the six Likert- Scale questions in the exit questionnaire.

Likert-Scale questions Agree Neutral Disagree

1. During the tele-health appointment I was able to see

the specialist clearly 43 0 0

2. During the tele-health appointment I was able to

hear what was said clearly 43 0 0

3. I felt comfortable having this appointment using the tele-health system

40 3 0

4. I could get the doctor to understand and respond to my concerns during the tele-health appointment

42 1 0

5. During the tele-health appointment I was worried that others might be listening

0 1 42

6. I felt that the face-to-face appointment was

necessary 13 7 23

Table 11.3: Responses from the patients and families to the six Likert-Scale questions

Questions 1 & 2 concerned technical aspects of the video and audio link. All 43