Questionnaire design
The study questionnaire was designed by the research team and in order to contribute to the quality of the instrument the lead investigator (CR) underwent questionnaire design training and the questionnaire was included in the peer review; following feedback the questionnaire was amended. This helped to contribute to the quality of the instrument and create an instrument using widely recognised ‘best practices’ (Artino et al. 2014).
Google Forms was used for questionnaire distribution, it was chosen because it could support all of the question and could be easily delivered. Data could also be easily managed. The questionnaire contained six demographic questions plus 24 pre-VP questions and 34 post-VP questions (Figure x and Appendix V.). Demographic questions included basic information about the respondent this provided the vital information required for a detailed analysis.
95
Figure 5.2. The participant's journey when completing the questionnaire. The process took place remotely via a single Google Form
After completing demographics questions and some preliminary questions on self-reported ability on various aspects of NOACs and AF (this area of the questionnaire was based on the NICE guidelines for anticoagulant counselling (National Institute for Health and Care Excellence 2014)) the respondents were presented with an embedded URL link to the VP application, and they were free to use the VP for as long as they wished, around 20 minutes was suggested. Respondents then returned to their questionnaire and completed a series of post-VP questions, this included identical self-reported ability questions as pre-VP.
Other questions were mainly Likert scales using four different scales, there was one ranking question that asked respondents to rank five options from the most to least relevant. There were also seven short-answer free-text qualitative questions, these questions were based on common satisfaction questions from the literature to support further explanation of the Likert responses (Douglass et al. 2013, Taglieri et al. 2017).
Pre-VP
• Demographic questions
• Preliminary questions on VP use and practice
• Pre-VP self-reported NOAC counselling ability
Using the VP
• Participant access VP via a URL web link
• Participant instructed to return to the questionnaire once they have
finsihed using the VP
Post-VP
• Post-VP self-reported NOAC counselling ability
• Satisfaction questions relating to the technology of the application
• Free text questions on improvments to the VP
96
Responder literacy, or the demand on the respondent to understand the questions, was considered in the questionnaire design (Bowling 2005). As a consequence of the respondents all being HCPs, a high level of literacy was anticipated. Nevertheless, the questionnaire format, aesthetics, and wording were designed so that it could be understood by any of the
respondents. No specialised wording was used and the terms NOAC and AF were defined within the questionnaire. Motivation for participation was incorporated in the recruitment emails and PIS as it was made clear to potential respondents that they may personally benefit from taking part in the study as they would have access to a novel education tool that may have consequences for their NOAC counselling abilities.
The questionnaire used 5-point Likert scales. This use of scales with a central point has been the subject of much debate since respondents may naturally gravitate to the central point when they do not know an answer or do not want to select one from those provided (Rattray and Jones 2007, Scott and Mazhindu 2014). If a scale has no central point this forces an answer, which can be beneficial when the subject is sensitive or controversial. Conversely, people who are genuinely neutral are not able to select an appropriate option if a central point is absent (Scott and Mazhindu 2014). In this study as there was no reason to remove the central point 5-point scales were used. The questionnaire also used a minimum variety of Likert scales and asked simple questions as this can reduce cognitive load (Lietz 2010).
To reduce acquiescence bias (‘yes-saying’/an excess of positive results) (Bowling 2005), the questionnaire included questions with a mixture of positive and negative anchors so it was clear if respondents had repeatedly selected responses of the same axis, as described by Bowling (2005), and Rattray and Jones (2007). This included asking if respondents found the VP useful, comfortable and enjoyable but in reverse, asking if the application was difficult to use. This helped to identify any acquiescence bias as these sets of questions should, in most cases, have opposite results. This in turn contributes to internal reliability (Bowling 2005).
97
Questionnaire quality
It has been highlighted that there is a lack of considerations for the quality of evaluation instruments (Jabbur-Lopes et al. 2012) and quality considerations appear to be absent from many of the studies discussed in chapters 2 and 3. A number of quality considerations have been considered in this study and are discussed below.
Initial reliability testing of the instrument consisted of establishing content and face validity. A review of the questionnaire by people who are experts in the field contributed to addressing content validity (Artino et al. 2014). The research team as pharmacists had first-hand
knowledge of the construct of interest and they could advise on the questionnaire content and language from the point of view of a respondent; the peer-review also contributed to this. Subsequently, improvements to the questionnaire delivery were made so that it was easy to use, accessible and not overly time-consuming as this has been shown to negatively impact response rates (Rolstad et al. 2011).
The draft instrument was piloted, which aimed to test the design and processes of the study, and prepare for data analysis via exploration of which tests would be suitable for use (van Teijlingen and Hundley 2001, Artino et al. 2014). A convenient sample of pharmacists from the School of Pharmacy was drawn on, in keeping with the pragmatic approach to this research. To avoid contamination between the pilot and the main study data was managed separately and participants could only take part in either the pilot or main study (Leon et al. 2011).
Within the pilot, all parts of the questionnaire data were examined to explore different statistical options for analysis of the data. Broadly, the pilot used the same data collection and analyses as the wider study. Ten participants took part in the pilot phase and data were examined for acquiescence bias. A significant Cronbach alpha result (P=0.917 and 0.865 for pre and post-VP Likert’s respectively) demonstrated a high level of internal consistency which in turn suggested that the Likert scales were reliable to be used in the wider study (Tsang et al.
98
2017). The pilot data behaved as expected, suggesting that the questionnaire functioned as intended, and there was also a trend in the data which implied that the respondents understood the questionnaire in a similar way. This increased its reliability for further use. Face validity was also assessed (Tsang et al. 2017). This involved using direct participant feedback gathered from the pilot to identify improvements to the delivery, design and overall experience of using and completing the questionnaire.
Following the pilot, minor amendments to the questionnaire content were identified and made. These included grammatical and typographical errors which did not change the purpose or nature of the questions. One ranking question was amended to reverse the ranking scale and to include the correct number of outcomes. This was amended for data analysis purposes and to make the question easier to understand for the respondents..
Data analysis
Questionnaire responses were accessed in Google Forms and converted to a Microsoft Excel spreadsheet for analysis. The data underwent descriptive analysis incorporating a mixture of SDs and means, and medians and IQR. These were used to establish the nature of the data and its common features and make data interpretable (Scott and Mazhindu 2014).
Further statistical analysis was then undertaken to analyse specific parts of the data. The study collected a mixture of ordinal and nominal data and included both parametric and non-
parametric tests (Scott and Mazhindu 2014). In this study, Likert data were analysed using a mixture of parametric tests and means and medians. This depended on whether data were normally distributed and the intended purpose of the question(s) as described below.
During analysis, the questionnaire was split into five sections. Demographic questions were first considered using descriptive statistics to establish and explore the nature of the sample. This was also particularly important for interview sampling. Self-measured ability Likert scales pre- and post-VP were then considered. The Likert items were grouped together into pre- and
99
post-VP groups, the cumulative and average score for each was calculated, as well as the percentage change. The convergence of multiple scores into one overall score is a recognised method of managing Likert data (Boone and Boone 2012). The grouped data then underwent statistical analysis using paired t-tests to establish whether there was a significant difference between the two sets of scores (Healey 2014). T-tests are a common test and have been widely used, including in VP evaluations (Battaglia et al. 2012, Al-Dahir et al. 2014). After statistical consultation and considering the nature of the data, t-tests were deemed appropriate as the data was normally distributed. A Kolmogorov-Smirnov test of normal distribution was used to aid this decision making.
The satisfaction questions were analysed using descriptive statistics, specifically calculations of medians and IQRs. Medians were used over means for these questions as the data was ordinal and there was no reason to treat it as interval data. A ranking question with five outcomes was analysed using a Friedman Test that identified whether there was a statistical significance between the responses (Field 2013).
The final questionnaire section consisted of the qualitative free-text questions, these responses were thematically grouped, and the frequencies calculated. This was used to support, explore, and confirm numerical data and also contributed to interview sampling.