81 4.1 Measuring the content validity of the WIMAT colonoscopy suitcase: Introduction a) Content validity
Content validity is the first stage of any simulation validation process (Table 5). This is “the extent to which a measure reflects the trait or domain it purports to measure” (Sedlack, 2011). Content validity is frequently measured using questionnaires for expert users to measure the haptic and visual realism of a simulator (Table 5). The aim of this study was to test the content validity of the WIMAT colonoscopy suitcase.
4.2 Methodology
All participants were recruited from regional and national endoscopic training courses where the simulator was being demonstrated. These were all faculty members, all highly experienced in colonic polypectomy. Participants were either independent Consultant practitioners and senior trainees or nurse endoscopists that were JAG accredited to perform colonoscopy. Each participant completed a snare polypectomy on a simulated pedunculated (bleeding/ non-bleeding) and sessile polyp. All polyps, equipment and endoscopic equipment were standardised throughout. Following the simulation, each participant completed a pre-designed realism questionnaire (Figure 29)
a) Construction of an expert realism survey to test content validity
The systematic review (chapter 2) demonstrated that there is only one study which reports the content validity of an ex-vivo colonoscopy simulator (Sedlack et al., 2007b). In this study, Sedlack et al focus on the complete validation of a bovine model for colonoscopy training (Sedlack et al., 2007b). A Mayo Clinic Survey (Mayo Clinic, Rochester, MN) was used to grade different aspects of the bovine model’s realism and complexity compared to a live
82 human colonoscopy (Figure 30). This validated survey was adapted to evaluate the content validity of the WIMAT colonoscopy suitcase (Figure 28). Questions 1-13 of the survey were divided into the following three areas: visual realism, anatomical realism, mechanical realism. Individual parameters within each of the 4 sections were scored on a 7-point Likert Scale (1 = strongly disagree, 4 = neutral, 7 = strongly agree). Question 14 focussed on the overall degree of similarity between the simulated polypectomy and “real life” polypectomy (1 = strongly disagree, 4 = neutral, 7 = strongly agree). Question 15 compared the technical difficulty of human polypectomy with the simulation (1 = much easier, 4 = same, 7 = much more difficult). The full questionnaire used during the study can be found in Appendix 2a.
b) Data analysis and power calculations
Assuming that the 7-point scale has a standard deviation of 1.0, it was calculated that 17 participants would give >90% power to detect a difference of 1 point or more on the survey against a hypothetical mean of 4. Realism surveys were analysed using the Wilcoxon signed-rank test on a PASW Statistics 18 (SPSS) for non-parametric data. Median values from the 7-point scale were compared with a hypothetical mean of 4 to determine statistical significance.
Demographic data is presented as mean values with 95% confidence intervals**.
** Statistical analysis has been reviewed and approved by Professor P Price
83 Figure 28: Questionnaire used for evaluating the content validity of the WIMAT
colonoscopy suitcase adapted from (Sedlack et al., 2007b)
84 Figure 29: Mayo Clinic Survey (Mayo Clinic, Rochester, MN) (Sedlack et al., 2007b)
85 4.3 Results††
a) General Results
A total of 17 participants (Male: Female ratio 14:3) completed the questionnaire; 15 (88.2%) Gastroenterologists, 1 (5.9%) Colorectal Surgeon and 1 (5.9%) experienced endoscopic Nurse Specialist. Of the Gastroenterologists, 7 (46.7%) were Consultants, 8 (53.3%) were ST6-7 level or Post Certificate of Completion of Training (CCT) fellows (Table 13). All participants were experienced in performing colonoscopy, polypectomy and biopsy. The mean numbers of previous procedures performed by the cohort were 371 (179-689) colonoscopies, 156 (35-355) polypectomies and 165 (42-360) biopsies (Table 14). The majority of the cohort had previous experience of using several different polypectomy simulators (Table 14).
b) Results of the realism survey
The WIMAT colonoscopy suitcase demonstrated a good level of realism scoring across a range of parameters. The highest scores were for “mucosal realism” (Median score 6.0, p=0.001), “endoscopic snare control” (Median score 6.0, p=0.001), “handling the polyp”
(Median score 6, p=0.001) and “raising mucosa” (Median score 6.0, p=<0.001) (Table 14).
Six parameters scored a median score of 5 with statistically significant results (Table 15).
These were “endoscopic view” (p=0.001), “polyp realism” (p=<0.001), “bleeding realism”
(p=0.013), “haustral folds” (p=0.029), “anatomical realism of pedunculated polyps”
(p=0.010), and “diathermy of the polyp” (p=0.026) (Table 15). Of the 15 parameters examined, only 3 were not statistically significant in favour of the simulator. These were,
“anatomical realism of sessile polyps” (p=0.080), “resistance of scope movement” (p=0.406) and “paradoxical motion” (p=0.055).
†† Full data set can be found in Appendix 2b
86 Clinical Level of participant Number of participants
Consultant 8 (47.1%)
Post CCT 2 (11.8%)
Senior Trainee 6 (35.2%)
Nurse Specialist 1 (5.9%)
Type of procedure Mean no. of real life procedures (95% CI)
Colonoscopies 371 (179-689)
Polypectomies 156 (35-355)
Biopsies 165 (42-360)
Type of simulation Mean no. of times simulator used (95% CI)
Virtual Reality 64 (3-180)
Animal Model 4 (1-9)
Bench Model 8 (1-19)
Table 14: Demographic data from participants (CCT = completion of clinical training)
c) Overall score
The overall score for the simulation was statistically significant when compared to a neutral score (Median score 6.0, p <0.001). When participants were asked to compare the level of difficulty of the simulator compared to real life the result was not statistically significant (Median score 4, p=0.559). This indicates that a high degree of correlation between the simulator and real life experience.
87 Table 15: Results from realism survey analysed using the Wilcoxon signed-rank for non-parametric data. Median values from the 7-point scale were compared with a hypothetical mean of 4 to determine statistical significance (significant parameters highlighted in grey) 4.4 Discussion
The results of this study have demonstrated that the WIMAT colonoscopy suitcase has a good level of content validity across a range of parameters. A cohort of participants, experienced in the skill of colonoscopy and polypectomy, awarded the model favourable scores for visual, anatomical and mechanical realism. All of the measured parameters for visual realism scored
88 well enough to produce a statistically significant result. Most encouragingly was the statistically significant score for the overall realism of the model and the non-statistically significant score when comparing the difficulty of the simulation to actual reality.
a) Non-statistically significant scoring
There were 3 parameters which did not reach statistical significance (Table 15). For anatomical realism, it was interesting to note that the scoring for simulated pedunculated polyps was more favourable than that for the simulated sessile polyps. However, when participants were asked to comment on the realism of performing a mucosal lift on a sessile polyp, a statistically significant favourable result was achieved. This would imply that the reduced level of anatomical realism of the sessile polyp did not significantly impact on the process of performing the polypectomy. The other non-statically significant parameters were for “resistance to scope movement” and “paradoxical motion”. This may be a limitation of the current model. However, this should not significantly affect the use of the simulator which is designed to focus on polypectomy training as opposed navigation and endoscopic steering.
b) Conclusion
The WIMAT colonoscopy suitcase demonstrates a good level of content validity
Experts scored the model favourably in terms of visual, anatomical and mechanical realism using a previously validated survey
Further work is needed to improve the anatomical realism of the simulated sessile polyps
89