Validation phase 1 methodology for simulator assessment

Assessment of surgical performance is a moot point. It is not only a final and evaluative part of the instructional unit. The assessment process occurs throughout the activity. The situation is complicated by the user’s need to perform clinical duties. Clinicians were only available for one-to-one demonstration and review of the system, with either on-line or paper-based questionnaires to complete. The solution to this was to recruit a trial manager, Dr Craig Reston MRCS, who was able to dedicate the time to the individual user within their work schedule.

Typical virtual environment topics for evaluation:This method may be used to assess any system, from a stand-alone desktop system that is designed to train a predominantly visual skill, to one based upon a distributed architecture as demonstrated here. This can be evaluated when performing the necessary skills of surgery. The key issue is not whether to creep forward through evolution of digital substitutes for other education, training and archive technologies, but whether to promote the revolution of clinical practice through the integration of pervasive computing technologies. By using both ‘Just-in-time’ as well as ‘Just In Case’ educational strategies, providing tools that deliver anonymised patient-specific data to the user in a virtual environment, it is potentially possible to seamlessly integrate this by virtue of an educational pedagogy. People need to strive for the vision (61).

Such early prototypes are largely being superseded; however, virtual environments can be updated easily and automatically. The move is towardautonomous computing. The use of a gaming on-line environment can be employed to allow users to run rendered wire frame models on their local machines but the other components can be delivered by store-and-forward services prior to the start of the training session. The opportunity is also there for feedback upon different versions of software, using such systems.

4.7.1 Educational environment generation for evaluation

This needs to meet the basic educational demands of instructional design with clearly stated goals and objectives. The methods of feedback and remediation are of interest as they pose more technical demands than the other aspects that are predominantly text-based. The following issues were addressed:

1.Conduct assessment of the trainee performance: in terms of a quantitative notation. Ideally this will model the actual track of the user for replay and discussion, but this is beyond the capabilities of the third generation EVW prototype.

2.Provision of feedback: right/wrong feedback with justification and relation to the subjacent theory and/or recall of the related theory and/or the visualisation of the result of the user’s process. At present the aforementioned supervisor provides this feedback. This will ultimately be based upon the user tracks referred to above.

3.Provision for remediation: redirection to other steps of the process – This is managed by bidirectionality of the streamed video.

4.Redirection to a prerequisite sectionor recall of the related theory – This is managed by links using hypertext links embedded within the video sequence, directly linking to the library components.

4.7.2 Internal structure and content of the educational scenario

This is the framework for constructing Multimedia Educational Orthopaedic Module checklists. These may be added by a tutor, as in the test scenario, when issuing the directive reading, with explanatory text for that specific course or outlook upon the generic text of the handbook. The handbook provided to the trainees does not closely follow all the guidelines of the surgical course model. The guidelines are in accordance with the structure, though the ‘trainee’ requirements section is not. Evaluation of the instructional event for the trial relating to the arthroscopic shoulder operation is found at: http://voeu.ecs.soton.ac.uk/ : VOEU : Library : Shoulder : Instruction : in three documents entitled: introduction, body, conclusion. The aims of these sections are:

Introduction

The purpose is to inform trainees of the purpose of the instructional unit, and of the professional interest of this. It should provide overview and description of how to use the instructional unit. It also provides explicit required background for user profile (professional issues) and prerequisites (knowledge issues).

Body

Information is presented in accordance with the chronological structure of the presented surgical intervention. At each step of the chronological structure information is presented according to the Mayer’s Multimedia Principle (88).

 Present information in different settings using text and diagrams or illustrations to detail the equipment, anatomy & recall related theory.

 Provide interactivity, i.e. opportunities for the trainee to test out and try understandings in some way – to act using gestural (gesticulation or dexterity)18 _and intellectual skills

 Provide feedback to the trainee’s actions: This includes explicit expert’s criteria for validation (“how can the surgeon decide that each step was achieved correctly”), by presenting information in different settings. Explicit specific terminology (definitions) should be described and right/wrong feedback given to responses, training users to recall related theory. This will provide a different setting for viewing the result of the user’s action.

Conclusion

This provides a summary of the instructional unit in terms of learning objectives “now you should be able to...” and provides further reading, whilst restating motivation aims. To adapt this generic approach for the shoulder module scenario, it is possible to build a specific checklist for the purpose. The principles should however be adhered to for comparison between simulation types.

4.7.3 Technical requirements

The following requirements were met for local alpha-testing by providing a catalogue of key components and requirements. This included a browser add-on requirement list, and a list of files and functions needed to support the system component. The Reviewers Guide has evolved during the beta-testing into the ‘Help Guide’linking into the ICT handbook. An underpinning RCS Educational Strategy has been written to provide a foundation for this.

Factors used to evaluate the technical system act as instruments for developing future surgical educational systems. They can be grouped as:

 Assigning points to effort – Extension of the eSTEP scheme

 Time vs. effort need to be defined for certain subtasks

18

The first two steps (equipment and anatomy) are in place, but for the reasons given in section 1, dexterity and intellectual skills are not included. Again these can be added by

 Ratio (balance) of effort upon different subtasks is set up by the individual – each person can contribute material to the educational resources, i.e. ‘own a module’

 Length of time users spend upon the creation of the material

 Active as well as passive learning approaches

 Weighting system for time and value of effort expended on tasks and subtasks by Stakeholders

 Feedback upon values of contribution from various authors

4.7.4 Scenario generation for evaluation - Relating the educational scenario to the pedagogy

This relates to the userquestionnaire. The following pedagogical principles need to be addressed and are scored on a 5 point ‘visual analogue scale’ (VAS):

 Feedback during the activity (a series of steps) that any actions or thoughts were either right or wrong

 Opportunity for the problem solver to start a single step in an activity again

 Feedback during a single step, will not prevent the problem-solver from accessing the following steps of the activity

 Thresholding of the results so that the problem-solver cannot proceed to the higher-level activities until he has successfully completed the lower level activities.

 The information that the problem-solver must be given, obtained from the investigation over time, allowing reasoning to occur at every step of the activity (89).