Prospective, Randomized Assessment of Transfer of Training
(ToT) and Transfer Effectiveness Ratio (TER) of Virtual Reality
Simulation Training for Laparoscopic Skill Acquisition
Anthony G. Gallagher, PhD, DSc,
∗Neal E. Seymour, MD,
†
Julie-Anne Jordan-Black, PhD,
‡
Brendan P. Bunting, PhD,
§
Kieran McGlade, MD,
¶
and Richard Martin Satava, MD
||
Objectives:We assessed the effectiveness of ToT from VR laparoscopic sim-ulation training in 2 studies. In a second study, we also assessed the TER. ToT is a detectable performance improvement between equivalent groups, and TER is the observed percentage performance differences between 2 matched groups carrying out the same task but with 1 group pretrained on VR sim-ulation. Concordance between simulated and in-vivo procedure performance was also assessed.
Design:Prospective, randomized, and blinded.
Participants:In Study 1, experienced laparoscopic surgeons (n=195) and in Study 2 laparoscopic novices (n=30) were randomized to either train on VR simulation before completing an equivalent real-world task or complete the real-world task only.
Results:Experienced laparoscopic surgeons and novices who trained on the simulator performed significantly better than their controls, thus demonstrat-ing ToT. Their performance showed a TER between 7% and 42% from the virtual to the real tasks. Simulation training impacted most on procedural error reduction in both studies (32- 42%). The correlation observed between the VR and real-world task performance wasr>0·96 (Study 2).
Conclusions:VR simulation training offers a powerful and effective platform for training safer skills.
Keywords: Operating room (OR), simulation, transfer of training (ToT), transfer effectiveness ratio (TER), virtual reality (VR)
(Ann Surg2013;257: 1025–1031)
T
raditionally, surgical skills were acquired in vivo with one-on-one learning in the operating room under the supervision of a master surgeon. This approach to learning has been considerably challenged1 by a sea-change in resident duty hours in the United States2and in the European Union.3Satava4proposed virtual reality (VR) simula-tion as an alternative strategy for the acquisisimula-tion of skills that would also minimize risk to patients, particularly during the early part of the learning process. Although widely used in the aviation,5nuclear,6 military7, and space sectors8for training high-risk, high-skilled per-formance characteristics, procedural-based interventional medicineFrom the ∗School of Medicine, University College Cork, Cork, Ireland;
†Department of Surgery, Tufts University School of Medicine, Springfield, MA; ‡School of Psychology, Queen’s University, Belfast, Northern Ire-land, United Kingdom; §School of Psychology, University of Ulster, Derry, Northern Ireland, United Kingdom;¶School of Medicine, Dentistry and Biomedical Sciences, Queen’s University, Belfast, Northern Ireland, United Kingdom; and ||Department of Surgery, University of Washington Medical Center, Seattle, WA.
Disclosure: The authors declare no conflicts of interest.
Reprints: Anthony G. Gallagher, PhD, DSc, School of Medicine, Brookfield Health Sciences Complex, College Road, University College Cork, Ireland. E-mail: anthonyg.gallagher@btinternet.com.
CopyrightC 2013 by Lippincott Williams & Wilkins ISSN: 0003-4932/13/25706-1025
DOI: 10.1097/SLA.0b013e318284f658
had almost no experience of this approach to training. Indeed, one of the major problems that interventional medicine had/has to over-come is the belief that surgical/procedural skills can only be learned while working on real patients. Other high-risk, high-skills sectors have developed an approach to learning practical skills that is based on an approach to learning developed by Edward L. Thorndike and Robert S. Woodworth9at the start of the 20th century. Thorndike and Woodworth proposed that what individuals learned in one context, they transferred it to another context that shared similar character-istics. Their theory implies that transfer of training (ToT) depends on the proportion to which the learning task and the transfer task are similar. In the aviation industry, to measure ToT from a flight simulator to an airplane, at least 2 groups of trainees are required. Speed of learning in an airplane by a group first trained in a sim-ulator is compared with the speed of learning by a control group trained only in the airplane.10Quantitative measures of transfer in-clude “transfer effectiveness ratio (TER),” which is the proportion of the time or trials saved in the airplane to the time or trials spent in the simulator. Transfer of training research (independent of simulation) has developed into a multibillion-dollar industry, producing a wealth of empirical findings and theoretical interpretations, which are often contradictory and frequently based on survey data.11,12
Despite VR simulation having a presence in surgery for almost 2 decades, its acceptance by the medical profession as a legitimate and powerful training technology is still tenuous. A small number of studies have shown that VR training improves operating room performance.13–17However, medicine has been critical of these stud-ies for a variety of reasons: the small number of subjects used, the fact that they have usually been conducted within a single institution, that the trainees in these studies were relatively junior (ie, surgical residents), and that the surgical procedure they were being prepared for was relatively low-risk, ie, laparoscopic cholecystectomy.15
The first aim of the studies reported here is to assess a number of aspects of the ToT effect—to replicate the ToT effect but with a group of very experienced laparoscopic surgeons. The second aim is to quantify the proportion of skills transferred for performance of a relatively straightforward but novel laparoscopic task from a VR simulation to an equivalent novel real-world task, ie, TER. In Study 2, the real-world task was built specifically to closely resemble the VR-simulated training task. The VR task is a component task of one of the best-validated VR simulators in medicine, ie, minimally invasive surgical trainer virtual reality (MIST VR). This simulator has been shown to have robust psychometric assessments that demonstrate con-struct, concurrent, discriminative, and internal validity.18–21It also demonstrated predictive validity in prospective, randomized clinical trials that showed that training on MIST VR improved intraopera-tive performance.13,14,16,22We will assess the concordance between performance on the simulation and the corresponding performance characteristics of performance on the real-world task. These data are important to give better insight into the relationship between the trainee’s performance on the VR simulation and his/her performance on the equivalent real-world task. A study examining equivalent r
In Study 1, numbers 1–210 and in Study 2, numbers 1–30 were assigned to one of two groups using a research number randomizer (by AGG). In each study, the number and their group allocation were kept in an envelope and retrieved (from an independent third party) just before subject participation in the study. Subjects in Study 1 were recruited by NES and RMS and in Study 2 by JAJ. Participants were assigned and trained by support researchers. Outcomes assessments and analysis were conducted by individuals blinded to participant identity and training group status. The flow of subjects through the study procedures are shown in Figure 1.
It is hypothesized that training in the VR environment transfers to real-world tasks and that this ToT is greater than 10%. In both studies, training transfer is assessed by having the 2 matched groups complete very similar world tasks. Before completing the real-world task, one group participated in a period of training on a VR simulator. Assessment of the TER effect was the difference between the objectively assessed performances of the VR-trained subjects in comparison with the non-VR-trained subjects on the real-world task. H0was that there would be no difference in performances between
the 2 groups in either of the 2 studies.
STATISTICAL ANALYSIS
Differences between groups were compared for statistical sig-nificance with 1-factor analysis of variance (ANOVA).
Study 1 Subjects
One hundred ninety-five experienced laparoscopic surgeons (ie, had performed more than 50 minimally invasive surgical proce-dures) from 13 countries participated in this study. Their demographic details are shown in Table 1.
Apparatus
Box-Trainer.A standard laparoscopic cutting task was used, as previously reported (Fig. 2A).23–25 Briefly, a cut is made in 26 spaces clearly demarcated by black lines along the long edge of a sheet of paper. The task was placed in a portable laparoscopic trainer (3-D Technical Services, Franklin, OH). Ethicon nontoothed scopic forceps (Ethicon Endosurgery, Cincinnati, OH) and laparo-scopic curved scissors were used to hold and cut the paper. This task has been shown to be sensitive to differences between experienced and junior surgeons, is sensitive to learning, measures correct responses and errors, has low variability, and has been demonstrated to conform to the Gaussian distribution and a high inter-rater reliability.23–26 MIST VR
The MIST-VR (Fig. 2B) system (Mentice AB, Gothenburg, Sweden) has been previously described by Wilson et al and others.27,28 It offers 6 tasks of incrementally greater complexity with 3 difficulty levels. Each one is based on a key surgical technique employed in laparoscopic cholecystectomy. Performance metrics are generated for each task formatively and summatively. A novel task based on the box-trainer task described above was developed for this study and is shown in Figure 2C. The task required the subject to hold a virtual
Subjects were prospectively randomized to complete the VR task first and then the trainer cutting task or they completed the box-trainer task only. The VR task required subjects to hold the virtual card with one instrument and excise the spheres by grasping them in the center with the other instrument. When this was done correctly, the sphere disappeared. After all 5 spheres had been excised, a new card appeared. All subjects were required to complete 4 cards with a total of 20 spheres. Timing was stopped between all the spheres being excised for 1 card and a new card appearing.
The laparoscopic cutting task was placed horizontally in a conventional box-trainer under standardized laparoscopic conditions. Subjects were required to make 1 incision between 26 spaces clearly demarcated by black lines along the long edge of the sheet of paper (US Letter). Subjects grasped the paper with their nondominant hand and cut with their dominant hand. An error was judged to have been made if a subject’s incision cut across one of the black lines or touched it. If an incision was so close to the line that the experimenter found it difficult to make a decision, it was judged to be an error. Twenty percent of sheets were rechecked by another investigator. The inter-rater reliability was 98% and the Cronbach alpha was 0.997. Subjects were asked to make as many incisions as they could in 2 minutes. The laparoscopic task was removed from the box-trainer after each subject was finished. Timing commenced when subjects placed the paper between the jaws of the scissors for their first incision.
Results
There were no significant differences in demographic charac-teristics between the 2 groups. The vast majority (in both groups) of surgeons who were assessed were from the United States. Both groups were very experienced laparoscopic surgeons and reported having performed more than 900 minimal invasive surgical (MIS) procedures during their career and more than 120 annually. The sur-geons who practiced the simple cutting task on the VR simulator before performing the real-world task, on average, made significantly more correct incisions on the real-world task (20.73 vs. 19.27;F (df=1, 193)= 4.717,P=0.031), ie, 7% more correct incisions (Fig. 3A). They also made a significantly lower percentage of incor-rect incisions (1.79 vs. 2.62;F(df=1, 193)=4.491,P=0.035), ie, 32% fewer incorrect incisions (Fig. 3B).
Study 2 Subjects
Thirty participants (15 female and 15 male) with no previous experience of laparoscopic surgery were recruited to the study. The mean age was 24.16 years (range 17–40 years). Fourteen of the par-ticipants reported corrected vision and all were right-hand-dominant.
Apparatus
MIST VR.The MIST VR system is as described above and shown in Figure 2B. The VR task is shown in Figure 4A and required subjects to grasp a virtual sphere with one instrument, transfer it to the instrument in the other hand and then place the sphere in the virtual box without touching the outer edges of the wire frame or releasing it before being accurately placed in the wire frame/box. The
FIGURE 1.Flow of trainees through Studies 1 and 2.
TABLE 1.Demographic Details of the Surgeons Tested
Age % of Surgeons VR 1st,N BT,N Country of Practice N % of Sample
30 – 39 years 24.6% 25 23 India 3 1.53
40 – 49 years 40.0% 36 42 Brazil 6 3.06
50 – 59 years 23.0% 22 23 Canada 15 7.65
60 – 69 years 12.4% 14 10 Colombia 3 1.53
70 years 0.0% 0 0 France 3 1.53
Total no. surgeons 97 98 Germany 3 1.53
Iceland 3 1.53
% n n Mexico 6 3.06
Right-hand dominant 92.3% 87 93 South Africa 3 1.53
Male 86.0% 79 89 USA 122 62.22
Venezuela 9 4.59 Mean no. career MIS procedures 903 (SD=1107) 1057 (SD=1003) UK 16 8.16 Mean no. annual MIS procedures 129 (SD=111) 152 (SD=89) Pakistan 3 1.53
Total: 195
BT=box trainer, VR=virtual reality.
FIGURE 2.The box-trainer cutting task (A) used as the test task; the minimally invasive surgical trainer virtual reality (MIST VR) setup (B), and the VR task (C) that was used to train subjects.
training required subjects to perform the task with alternate hands, thus training performance ambidexterity.
Box-Trainer.A cellular target area (170 mm×170 mm×28 mm) containing 12 boxes with a 4×3 matrix (each box 35 mm× 52 mm×28 mm) was placed at a 45-degree angle in a conventional
box-trainer shown in Figure 4B. The target area was viewed at a right angle through a Panasonic miniature camera (model no. WV KS 152) that was capable of producing on a Panasonic 14-inch color monitor (model no. TC-14SIR). The task was maintained in the center of the visual field. Ethicon nontoothed laparoscopic forceps were used
FIGURE 3.The mean (and standard error) num-ber of correct (A) and the percentage of in-correct incisions (B) made by the untrained
(n=97) and the VR-trained (n=98) surgeons.
FIGURE 4.The experimental setup for performing the VR acquire, transfer, and place task (A), and the real-world box-trainer task that duplicated the VR task (B). Figure 4C demonstrates that the goal of the task in setup A (VR) and B (real world) were the same, ie, transfer the sphere from one instrument to the other and then place it in the box. An error was scored if the instruments holding the sphere touched the outer edges of the box. In the VR version, an error was scored if the sphere was not released by the instrument when placed in the box, and in the real-world version, if the sphere was dropped outside the target box. to “grasp” and “manipulate” circular sponge targets (diameter 12
mm). The forceps were linked through a processed timer to a digital timer counter (made by Electronic Developments). The digital counter recorded the number of errors made (the number of contacts made by the instrument on the target area). Time was recorded using a stopwatch.
For both parts of the experiment, the monitor was positioned at eye level with the tools at standard surgical height between the participant and the monitor, as shown in Figure 2B.
Procedure
Each participant was randomly assigned to 1 of 2 conditions.
Condition 1
Participants completed a simulated task on MIST VR shown in Figure 4A. The VR task required subjects to acquire a virtual sphere with one instrument, transfer it to the instrument in their other hand and then place it in a virtual wire frame, without dropping the sphere or touching the outer edges of the wire frame with the sphere as it was released. This was repeated 5 times for each hand. Time and error scores were recorded by the simulator. Participants then completed a real-life task on a box-trainer that was similar to the simulated task on MIST VR (Fig. 4B). The task involved grasping a sponge sphere, transferring it between instrument tips and placing it in one of the boxes contained on the cellular target area. The experimenter indicated which box the sphere was to be placed in. Like the MIST
VR task, this also was repeated 5 times for each hand and similarly an error was scored and recorded by the digital counter if the subjects touched the walls or edge of the box. A stopwatch was used to record time and a digital timer counter recorded the number of errors made by the subject. The performance requirements for both the virtual and the real-world tasks in this study were very similar as demonstrated in Figure 4C.
Condition 2
Participants completed only the real-life task on the box-trainer (as described above). No previous training was given to this group.
Results
Figure 5A shows the mean time taken to complete the laparo-scopic task in the box-trainer by subjects who had previous training on MIST VR (Condition 1) and by subjects who only completed the task on the box-trainer and received no previous VR training (Condition2). Differences between the group means for time were examined using a 1-way ANOVA for independent groups design. Participants who had previously trained on MIST VR took significantly less time (ie, 26%) in completing the task on the box-trainer than those who had received no previous training (F(1, 28)=12.71,P=0.001). They also made 42% fewer errors (Fig. 5B) on the real-world task than subjects who had received no previous VR training (F(1, 28)= 7.69,P=0.010).
FIGURE 5.Mean and standard error scores for the number of seconds it took for the standard
(n= 15) and the VR-trained (n= 15) groups
to perform the “real-world” task (A) and the mean number of errors made while performing the task (B).
FIGURE 6.The time scores of the
simulation-trained subjects (n=15) on the VR task plotted
against their time scores on the real-world trainer task (A), their VR and real-world box-trainer task errors scores plotted against each other (B), and their regression curves with 95% individual prediction intervals.
The time taken by subjects in Condition 1 to complete the VR task and then the box-trainer task are shown on a scatterplot in Figure 6A. VR performance time was highly predictive of time to complete the real-world task, the Pearson product moment correlation coefficient ofr=0.964, which accounted for 92.4% (ie, adjustedr -squared=0.924) of the variance (F(1, 13)=170.23,P=0.0001). The number of errors enacted during VR performance of the task was also highly predictive of real-world task performance (Fig. 6B) where r=0.974, which accounted for 94.6% of the variance (F(1, 13)= 244.30,P=0.0001).
DISCUSSION
Both studies show that VR simulation training transfers to improved performance on real-world tasks. The lowest proportion of performance improvement was observed in Study 1. Experienced sur-geons made 7% more correct incisions after VR training. The largest proportion of performance improvement was observed for the laparo-scopic novices (42%) in Study 2. Might subjects who had been trained on the box-trainer assessment task have performed equivalent or bet-ter than the VR-trained group in Study 2? In a previous prospective, randomized, and blinded study, we addressed this question.20The first group (the controls) received only the assessment (as in Study 2), the second-group subjects weretrainedon the task they were to subse-quently be assessed on, and the third-group subjects were trained on the MIST VR. The amount of training that the second-group sub-jects had practicing the box-trainer test task was case-matched to the amount of time that the VR-trained group subjects took to
com-plete their training. We found that the VR-trained group significantly outperformed the control and standard trained groups.
Even though the performance improvement of the experienced surgeons in Study 1 was only 7% better than the control group, it was a statistically significant improvement. In contrast, the same group of surgeons demonstrateda 31% lower error ratein comparison with the control group. Both of these findings could indicate a ToT effect and/or an effect of procedure warm-up that has been shown to im-prove procedure performance.29The same pattern was also observed in Study 2, ie, VR training appeared to have its greatest impact by lowering the actual number of errors on the real-world task. This finding may be explained by the fact that both VR simulations specif-ically targeted the enactment of intraoperative errors during training. In both VR training simulations, trainees received immediate noti-fication by the simulator after each error. This type of (formative) feedback allows for optimal learning due to the configuration of per-formance feedback during practice versus after a task is completed. Furthermore, data from Study 2 showed that performance on the VR task was highly predictive of performance on the real-world tasks. Study 2 probably gives the most robust estimate of TER for skill acquisition by novices. Although the task experienced surgeons were required to perform in Study 1 was novel to them, they had already acquired and honed their MIS surgical skills over many in-vivo pro-cedures. Thus an improvement in their performance of only 7%, even though statistically significant, was not surprising. What was surpris-ing was the considerable decrease that previous VR trainsurpris-ing had on error enactment (ie, 32%). For the novices, a rate of 26–42% TER30is considerable, particularly for skills at the start of a trainee surgeon’s
FIGURE 7.Graph showing the probability of a bile duct injury as a function of surgeon experience with laparoscopic cholecystec-tomy (LC) [30]. It shows that the probability of a bile duct injury during the first 10 cases performed by the trainee was approxi-mately 1 in 50, which had decreased to approxiapproxi-mately 1 in 500 by their 50th case. The shaded area indicates the proportion of the learning curve that would be eliminated based on the data presented in this article (ie, from Study 2).
learning curve where the patient is at the greatest risk of an error from the trainee’s performance.
Figure 7 shows the real-world implications of 26–40% TER to clinical performance. It shows extrapolated data from the Southern Surgeons Club,31 which was one of the earliest studies to quanti-tatively demonstrate that the risk of a serious complication (ie, bile duct injury) during a laparoscopic cholecystectomy was directly asso-ciated with the operative experience of the trainee. The graph shows that during the first 10 laparoscopic cholecystectomies performed by the trainee, the risk of a bile duct injury is almost 1 in 50, whereas the risk at their 50th procedure had decreased to about 1 in 500. The graph also shows the impact of a modest 26% TER and a 42% ToT; both rates of transfer (if our hypothesis is correct) would eliminate the early part of the learning curve when the risk is the greatest to the patient. Thus, training on the simulator supplants this early learning experience, immaterial of procedure,assumingsimulation training effectiveness. It should also be noted that the results reported here refer purely to a ToT effect, and whether the same findings would be observed for training to proficiency now needs to be investigated. We would however expect similar positive results.
Simulation models for skills training are effective because they provide a context, an organizational structure, and focus to apply in-formation retrieved from long-term memory.32–34Trainees can also practice the sequencing of psychomotor skills to complete the task effectively, efficiently, and safely with proximate highly structured formative performance feedback. Thus, trainees can learn and re-hearse what to do and with which instruments. It is also important that they learn what not to do. From the evidence presented here and reports from clinical studies on VR simulation training,13,14,16,17,35–37 it would appear that simulation training has its greatest impact on fa-cilitating a reduction in performances that deviate from optimal task execution. Furthermore, the simulation training does not have to be virtual. In a study on advanced laparoscopic skill acquisition (ie, in-tracorporeal suturing for Nissen fundoplication), it was shown that training surgical residents to proficiency38on improvised simulation models significantly improved intraoperative performance on Nissen fundoplication.39Simulation models provide a vehicle for the con-figuration and delivery of a teaching and training curriculum and an essential part of this curriculum is performance feedback. Metric-based performance feedback is preferable because it is objective, transparent, reliable, and precise.33 Although the simulation train-ing can be undertaken on a bespoke model, it is preferable that the simulation model is virtual because of the automated nature of
per-formance assessments that this affords. It has been demonstrated that bespoke simulation models are effective training platforms; however, the investment required from trainers to provide proximal formative performance feedback over an extended period of time to trainees is considerable.39–41
Indeed this is one of the limitations of this study. Although we have demonstrated ToT and provided estimates of TER based on prospective, randomized experimental data, we have not provided an estimate of the cost implications. This is another metric for ToT that the surgical (and medical) community will require to help them make a business case for investment in VR simulation technology for training the next generation of surgeons. These trainee surgeons do not have the quantity of case exposure that previous generations of trainee surgeons had.42 This means that they must be optimally prepared for the case exposure that they do get, which requires efficient and effective skills laboratory training to a level of proficiency before operating in vivo under the supervision of a master surgeon.
CONCLUSIONS
VR simulation training transfers to improved performance on real-world tasks and the transfer effect appears to be the greatest on the reduction in errors made. These results mean that VR simula-tion training can be used to supplant the early part of the procedure learning curve, which means better and safer operations.
REFERENCES
1. Pellegrini CA. Surgical education in the United States: navigating the white waters.Ann Surg. 2006;244:335–342.
2. Beall DP. The ACGME Institutional Requirements: What Residents Need to Know.J Am Med Assoc. 1999;281:2352.
3. Pickersgill T. The European Working Time Directive for doctors in training. BMJ. 2001;323:1266.
4. Satava RM. Virtual reality surgical simulator.The first steps.Surg Endosc. 1993;7:203–205.
5. Salas E, Bowers CA, Rhodenizer L. It is not how much you have but how you use it: toward a rational use of simulation to support aviation training.Int J Aviat Psychol. 1998;8:197–208.
6. Webster CS. The nuclear power industry as an alternative analogy for safety in anaesthesia and a novel approach for the conceptualisation of safety goals. Anaesthesia. 2005;60:1115–1122.
7. King DR, Patel MB, Feinstein AJ, et al. Simulation training for a mass casualty incident: two-year experience at the Army Trauma Training Center.J Trauma. 2006;61:943–948.
8. Rolfe JM, Staples KJ.Flight Simulation: Cambridge: Cambridge University Press; 1988.
9. Thorndike EL, Woodworth RS. The influence of improvement in one men-tal function upon the efficiency of other functions. Psychol Rev. 1901;8: 247–261.
10. Roscoe SN, Williges BH, eds.Chapter 16: Measurement of Transfer of Train-ing. Ames, IA: Iowa State University Press; 1980:182–193.
11. Saks AM, Belcourt M. An investigation of training activities and transfer of training in organizations.Hum Resour Manage. 2006;45:629–648.
12. Baldwin TT, Ford JK. Transfer of training: a review and directions for future research.Personnel Psychol. 1988;41:63–105.
13. Ahlberg G, Enochsson L, Gallagher AG, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies.Am J Surg. 2007;193:797–804.
14. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236:458–463; discussion 463–454.
15. Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a sys-tematic review.Ann Surg. 2006;243:291.
16. Grantcharov TP, Kristiansen VB, Bendix J, et al. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91:146–150.
17. Larsen CR, Soerensen JL, Grantcharov TP, et al. Effect of virtual re-ality training on laparoscopic surgery: randomised controlled trial. BMJ. 2009;338:b1802.
18. Gallagher AG, Lederman AB, McGlade K, et al. Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance. Surg Endosc. 2004;18: 660–665.
19. Gallagher AG, Satava RM. Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc.2002;16:1746–1752.
20. Gallagher AG, Hughes C, Reinhardt-Rutland AH, et al. A case-control com-parison of traditional and virtual-reality training in laparoscopic psychomotor performance.Minim Invasive Therapy Allied Technol. 2000;9:347–352. 21. Pearson AM, Gallagher AG, Rosser JC, et al. Evaluation of structured and
quantitative training methods for teaching intracorporeal knot tying.Surg En-dosc. 2002;16:130–137.
22. Seymour NE, Gallagher AG, Roman SA, et al. Analysis of errors in laparo-scopic surgical procedures.Surg Endosc. 2004;18:592–595.
23. Gallagher AG, McClure N, McGuigan J, et al. An ergonomic analysis of the fulcrum effect in the acquisition of endoscopic skills.Endoscopy. 1998;30:617– 620.
24. Crothers IR, Gallagher AG, McClure N, et al. Experienced laparoscopic sur-geons are automated to the “fulcrum effect”: an ergonomic demonstration. Endoscopy. 1999;31:365–369.
25. Gallagher AG, Jordan-Black JA, O’Sullivan GC. Prospective, randomized as-sessment of the acquisition, maintenance and loss of laparoscopic skills.Ann Surg. 2012;256:387–393.
26. Gallagher AG, Smith CD, Bowers SP, et al. Psychomotor skills assessment in practicing surgeons experienced in performing advanced laparoscopic proce-dures.J Am Coll Surg. 2003;197:479–488.
27. Wilson MS, Middlebrook A, Sutton C, et al. MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance.Ann R Coll Surg Engl. 1997;79:403–404.
28. Sutton C, McCloy R, Middlebrook A, et al. MIST VR. A laparoscopic surgery procedures trainer and evaluator.Stud Health Technol Inform.1997;39:598. 29. Kahol K, Satava RM, Ferrara J, et al. Effect of short-term pretrial practice on
surgical proficiency in simulated environments.Randomized trialJ Am Coll Surg.2009;208:255–268.
30. Taylor HL, Lintern G, Hulin CL, et al. Transfer of training effectiveness of a personal computer aviation training device.Int J Aviat Psychol. 1999;9:319– 335.
31. The Southern Surgeons Club. A prospective analysis of 1518 laparoscopic cholecystectomies.N Engl J Med. 1991;324:1073–1078.
32. Anderson JR. Acquisition of cognitive skill.Psychol Rev. 1982;89:369–406. 33. Gallagher AG, O’Sullivan GC.Fundamentals of Surgical Simulation;
Princi-ples & PracticesLondon: Springer-Verlag; 2011.
34. Van Sickle K, Gallagher AG, Smith CD. The effect of escalating feed-back on the acquisition of psychomotor skills for laparoscopy.Surg Endosc. 2007;21:220–224.
35. Palter VN, Grantcharov T, Harvey A, et al. Ex vivo technical skills training transfers to the operating room and enhances cognitive learning: a randomized controlled trial.Ann Surg. 2011;253:886–889.
36. Sroka G, Feldman LS, Vassiliou MC, et al. Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room—a randomized controlled trial.Am J Surg. 2010;199:115– 120.
37. Prinssen M, Verhoeven ELG, Buth J, et al. A randomized trial comparing conventional and endovascular repair of abdominal aortic aneurysms.N Engl J Med. 2004;351:1607–1618.
38. Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training.Ann Surg. 2005;241:364–372.
39. Van Sickle K, Ritter EM, Baghai M, et al. Prospective, randomized, double-blind trial of curriculum-based training for intracorporeal suturing and knot tying.J Am Coll Surg. 2008;207:560–568.
40. Van Sickle K, Baghai M, Huang IP, et al. Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying. Am J Surg. 2008;196:74–80.
41. Van Sickle K, Smith B, McClusky DA, III, et al. Evaluation of a ten-siometer to provide objective feedback in knot-tying performance.Am Surg. 2005;71:1018–1023.
42. Bell RH, Jr., Biester TW, Tabuenca A, et al. Operative experience of residents in US general surgery programs: a gap between expectation and experience. Ann Surg.2009;249:719–724.
![FIGURE 7. Graph showing the probability of a bile duct injury as a function of surgeon experience with laparoscopic cholecystec-tomy (LC) [30]](https://thumb-us.123doks.com/thumbv2/123dok_us/443495.2551317/6.837.42.403.109.397/figure-showing-probability-function-surgeon-experience-laparoscopic-cholecystec.webp)
