[��i
r-:
DocuSigned by:�i!:L
SIGNATURE
SIGNATURE
U::�"}''a�Ju-s6�
SIGN�ftf:irnA
14 IE ...8/5/2020 DATE
8/6/2020 DATE
8/6/2020 DATE
DocuSign Envelope ID: AB1 FE97A-70DE-468E-B82D-760880DA95F9
CALIFORNIA STATE UNIVERSITY SAN MARCOS PROJECT SIGNATURE PAGE
PROJECT SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE
MASTER OF SCIENCE IN
BIOTECHNOLOGY
PROJECT TITLE: Evaluation of Psychomotor Skills Assessment
AUTHOR: Christina Dages
DATE OF SUCCESSFUL DEFENSE: July 30
th ,2020
THE PROJECT HAS BEEN ACCEPTED BY THE PROJECT COMMITTEE IN
PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN BIOTECHNOLOGY.
Dr. Betsy Read
PROJECT COMMITTEE CHAIR
Dr. Moorthy Palanki
PROJECT COMMITTEE MEMBER
Carrie Jo Anderson
PROJECT COMMITTEE MEMBER
Christina Dages July 2020 Evaluation of Psychomotor Skills Assessment Executive Summary
Evaluation of Psychomotor Skills Assessment Pima Medical Institute- San Marcos
Christina Dages April 2020
Professional Master’s Degree Program California State University San Marcos
This Semester-In-Residence Project was conducted at Pima Medical Institute (PMI) in San Marcos California for use within the Veterinary Technician Program. The goal of this project was to establish the need to implement standardized criteria for evaluation of instructor assessment of psychomotor skills. In order to do so, four psychomotor skills were chosen to be independently assessed by five instructors within the Veterinary Program at PMI’s San Marcos, CA campus. The skills assessed were 1) aseptic closed gloving technique, 2) intravenous (IV) catheter placement, 3) K9 intravenous injection, and 4) K9 intramuscular injection. Instructors were asked to virtually assess student performance from a prescreened list of five videos for each psychomotor skill, providing a pass or fail grade. Instructors were also asked to provide comments as to their reasoning for a particular grade. The pass or fail score and comments from each evaluator were then compared to assess continuity across instructors within the same institution.
Results from the study demonstrated a lack of continuity between instructors across multiple videos.
Statistical analysis using a chi square (χ
2) test for each of the psychomotor skills determined a significant difference between the instructor’s assessment. This ultimately means some students might earn a passing mark on a psychomotor skill per one instructor and fail based on the score of another evaluator.
However, there was similarity within instructor comments as to a particular reason for either passing or failing a student. For example, a student’s ability to maintain sterility or use of proper technique, etc.
Students in the program must successfully complete all required psychomotor skills to use them in a real- world clinical setting as an entry level veterinary technician. Discrepancies between individual instructor evaluation of psychomotor skills must be resolved at the level of the evaluator.
The importance of this data relates to students entering clinical settings with insufficient technical skills and therefore compromising patient safety. All medical professionals, regardless of profession, are expected to uphold the highest standards of patient care and safety. Inconsistencies across instructor assessment of required program psychomotor skills ultimately lowers this threshold, striking a need to standardize assessment methods.
It is recommended that PMI San Marcos’s veterinary program rewrite all rubrics to include specific
procedural steps to improve instructor assessment continuity. The rubrics should also convert the current
Christina Dages July 2020 Evaluation of Psychomotor Skills Assessment
pass or fail grading system to a more streamline system that includes categories in which instructors can score a student based upon the comprehension and success of attempt. Another recommendation of this study is to ensure that the newly formatted rubrics are written by subject matter experts that have a minimum of five years’ experience in a hospital setting and licensure as a Registered Veterinary
Technician (RVT) or Doctor of Veterinary Medicine (DVM). Further studies should be conducted across
other PMI locations and programs to compare evaluator continuity.
EVALUATION OF PSYCHOMOTOR SKILLS ASSESSMENT
Pima Medical Institute- San Marcos Christina Dages
July 2020
WHO IS PIMA MEDICAL INSTITUTE?
• Medical career college
• Hands on training
• Certificate to Bachelors degree programs
• complete in 1-2 years
• Eighteen campus locations across the US
• Veterinary Technician Program
• 78.5 credits= 1775 hours of combined lecture, lab, and externship
WHAT ARE PSYCHOMOTOR SKILLS?
• Psychomotor skills are hands on skills that allow the student to perform a task that is identical to those used in a clinical setting.
• Program required skills
• American Veterinary Medical Association (AVMA) Appendix I
• Patient care and safety
• Gold standard practices
• Methods of practice
• Demonstration by instructor, medical phantom, virtual reality
• Lab time used for practice under direct supervision
GOAL OF PSYCHOMOTOR
PERFORMANCE
Figure 1. Conscious
Competence Model of Skills Acquisition. Psychomotor skills start at the unconscious
incompetent level. The goal of practicing a skill is to
eventually reach the final phase
of unconscious competency
(Cannon, 2014).
PROJECT OVERVIEW
Examine current evaluation methods used for the assessment of student
psychomotor skills performance.
MATERIAL AND METHODS
• Five subject matter experts (SME)- PMI instructors
• Three licensed Veterinary Technicians (RVT)
• Two licensed Veterinarians (DVM)
• Four psychomotor skills
• 1.) Closed gloving technique
• 2.) IV catheter placement
• 3.) K9 IV injection
• 4.) K9 IM injection
• Twenty prescreened videos (virtual skills)
• Five per psychomotor skill
Figure 2. Pima Medical Institute rubric for aseptic closed gloving technique. Created for the use of Veterinary Technician programs (PMI, 2019).
Skill Date and Initials of
Evaluator (PASS/FAIL) Grading Rubric
Closed Gloving
Techniques as a Scrub Nurse
• The student picked up the first glove from the open package and placed it thumb toward the palm and fingers pointing towards the elbows.
• The student grasped the glove on both sides through the sleeves of the gown and stretched the glove to create an opening.
• The student pulled the glove over the hand.
• The student moved their fingers from inside the cuff to the glove.
• The student placed their fingers into the fingers of the glove.
• The student pulled the sleeve of the gown and adjusted the glove to fit.
• The student repeated the process with the other glove and hand.
• The student made final adjustments on the gloves so that they fit snugly on the hands.
• The student did not contaminate the gloves or gown while putting the gloves on their hands.
Pass: Able to perform the skill in a laboratory setting in a safe, professional, and effective manner with distant supervision. Student should be able to perform the skill as an entry level veterinary technician.
Fail: Unable to perform skill in a laboratory setting in a safe, professional, and effective manner without constant cues and direct supervision. Student requires remedial training.
Current Rubric for Closed Gloving Technique Assessment
Figure 3. Pima Medical Institute rubric for placement of a cephalic intravenous (IV) catheter. Created for the use of Veterinary Technician programs (PMI, 2019).
Skill Date and Initials of
Evaluator (PASS/FAIL) Grading Rubric
Perform placement of intravenous catheter- cephalic
• The student clipped an area of appropriate size, leaving no hair at the site.
• The student prepped the site for an aseptic catheter placement and did not contaminate the site once it was prepped.
• The student flushed the catheter with a heparinized saline prior to placement.
• The student placed the catheter into the skin with the bevel up.
• The student looked for blood flow into the catheter and when blood was observed, threaded the catheter off the stylet into the vein.
• The student removed the stylet from the catheter and placed an injection cap on the catheter.
• The student secured the catheter to the patient leg with tape without kinking the catheter.
• The student flushed the catheter with heparinized saline
Pass: Able to perform the skill in a laboratory setting in a safe, professional, and effective manner with distant supervision. Student should be able to perform the skill as an entry level veterinary technician.
Fail: Unable to perform skill in a laboratory setting in a safe, professional, and effective manner without constant cues and direct supervision. Student requires remedial training.
Current Rubric for IV Catheter Placement Assessment
TRUE OR FALSE GRADING ASSESSMENT
Closed Gloving
Video 1 Video 2 Video 3 Video 4 Video 5
Instructor 1 P P F P P
Instructor 2 F P F F P
Instructor 3 F P F P P
Instructor 4 F P F F P
Instructor 5 F P F F P
IV Catheter Placement
Video 1 Video 2 Video 3 Video 4 Video 5
Instructor 1 F F P F F
Instructor 2 P F P P P
Instructor 3 P F P P P
Instructor 4 P F P P F
Instructor 5 F F F F F
Table 2. Results of instructor assessment of psychomotor skill 2) IV catheter placement Table 1. Results of instructor assessment of psychomotor skill 1) closed gloving technique
*
P= pass and F= fail
*
P= pass and F= fail
TRUE OR FALSE GRADING ASSESSMENT CONT’D
K9 IV injection
Video 1 Video 2 Video 3 Video 4 Video 5
Instructor 1 P P F F P
Instructor 2 P P F P P
Instructor 3 P P F P P
Instructor 4 P P F F P
Instructor 5 P P F F P
Table 3. Results of instructor assessment of psychomotor skill 3) K9 IV injection
K9 IM injection
Video 1 Video 2 Video 3 Video 4 Video 5
Instructor 1 P P P F F
Instructor 2 F P P F P
Instructor 3 P P P F P
Instructor 4 P P P F P
Instructor 5 F P P F F
Table 4. Results of instructor assessment of psychomotor skill 4) K9 IM injection
*
P= pass and F= fail
*
P= pass and F= fail
ALL FOUR PSYCHOMOTOR SKILLS ASSESSED REFLECT GRADING DIFFERENCES
• Most grading differences seen in psychomotor skill 2.) IV catheter placement.
• Videos 1, 3, 4 and 5
• Least grading differences seen in psychomotor skill 3.) K9 IV injection.
• Video 4
MAJOR ASSESSMENT DISCREPANCIES
Table 5. Major discrepancies across instructor psychomotor skills assessment of closed gloving.
Closed Gloving
Video 1 Video 4
Instructor 1 Sterility was maintained &
concept understood Sterility maintained & concept understood
Instructor 2 Sterility not maintained Sterility not maintained
Instructor 3 Sterility not maintained Sterility maintained
Instructor 4 Sterility not maintained Sterility not maintained
Instructor 5 Sterility not maintained Sterility not maintained
MAJOR ASSESSMENT DISCREPANCIES CONT’D
IV Catheter Placement
Video 1 Video 3 Video 4 Video 5
Instructor 1 Aseptic preparation incorrect &
insufficient taping
Proper technique Inappropriate
catheter insertion Sterility not maintained Instructor 2 Proper technique Proper technique Proper technique Proper technique Instructor 3 Proper technique Proper technique Proper technique Proper technique Instructor 4 Proper technique Proper technique Proper technique Sterility not
maintained Instructor 5 Contact time during
aseptic preparation is insufficient
Contact time during aseptic preparation
is insufficient &
sterility not maintained
Aseptic preparation incorrect & sterility
not maintained
Contact time during aseptic preparation
is insufficient &
sterility not maintained
Table 6. Major discrepancies across instructor psychomotor skills assessment of IV catheter placement.
MAJOR ASSESSMENT DISCREPANCIES CONT’D
K9 IV injection Video 4
Instructor 1 Incorrect technique & unsuccessful attempt Instructor 2 Appropriate preparation & technique Instructor 3 Appropriate preparation & technique Instructor 4 Incorrect technique & unsuccessful attempt Instructor 5 Incorrect technique & unsuccessful attempt
Table 7. Major discrepancies across instructor psychomotor skills assessment of K9 IV injection
Table 8. Major discrepancies across instructor psychomotor skills assessment of K9 intramuscular injection.
K9 IM injection
Video 1 Video 5
Instructor 1 Appropriate technique Incorrect technique
Instructor 2 Incorrect technique Appropriate technique
Instructor 3 Appropriate technique Appropriate technique
Instructor 4 Appropriate technique Appropriate technique
Instructor 5 Incorrect technique Incorrect technique
CATEGORICAL THEMES AMONG MAJOR ASSESSMENT DISCREPANCIES
• Major categories concluded from instructor comments as shown in Tables 5-8 included:
• Sterility
• Technique
• Successful or unsuccessful attempt
STATISTICAL ANALYSIS
Chi Square ( 𝝌𝝌 2 )
Closed Gloving 16.786 IV Catheter Placement 16.786 K9 IV Injection 18.283 K9 IM Injection 15.272
Tables 9-12. Chi square results of all four psychomotor skills assessed (p>0.001).
DISCUSSION
• Need to standardize assessment methods
• Lack of instructor training coupled with insufficient rubrics
• Mock demonstration and simulation labs
• Role of psychomotor skills in academics
• Student vs instructor
DISCUSSION
• Three common discrepancies
• Sterility, technique, and success of attempt
• Compromise patient safety
• Suggestive of need for grading scale based on proficiency
• Current pass or fail system is insufficient
• Allows for personal judgement or bias
• Current rubrics (Figures 2 and 3)
• Include various methodologies
• Missing rubrics
Grading Category Criteria
1-Remedial The student is unsuccessful in the task and does not understand the concept. Needs to be reassessed.
2-Developing The student is successful in the task and minimally understands the concept. Requires moderate supervision. Needs to be reassessed.
3-Functional The student is unsuccessful the task but has a good understanding of the concept. Requires minimal supervision.
4-Proficient The student is successful in the task and has a good understanding of the concept. Requires minimal supervision.
5-Advanced The student is successful in the task and has an excellent understanding of the concept.
Requires no supervision.
Suggested New Grading Criteria
FINAL RECOMMENDATIONS FOR
IMPLEMENTATION OF STANDARDIZED CRITERIA
• Rewrite all rubrics to include specific procedural steps
• SME: 5 years experience in hospital setting with RVT or DVM licensing
• Include grading criteria and remove pass or fail system
• Implement mock demonstration labs for all current staff
• Further studies should be conducted
• In person and various campus locations
REFERENCES
1
AVMA. “CVTEA Accreditation Policies and Procedures - Appendix I.” CVTEA Accreditation Policies and Procedures - Appendix I, American Veterinary Medical Association, Jan. 2020, www.avma.org/education/accreditation/programs/cvtea- accreditation-policies-and-procedures-appendix-i.
2
Fadde, P. J. (2009). Training Complex Psychomotor Performance Skills. Handbook of Improving Performance in the Workplace:
Instructional Design and Training Delivery, 468-507. doi:10.1002/9780470587089.ch14
3
Gallagher, A., McClure, K., & McGuigan, J. (2001). Objective Psychomotor Skills Assessment of Experienced, Junior, and Novice Laparoscopists with Virtual Reality. World Journal of Surgery, 25(11), 1478-1483. doi:10.1007/s00268-001-0133-1
4
Gonzalez, L., & Kardong-Edgren, S. (2017). Deliberate Practice for Mastery Learning in Nursing. Clinical Simulation in Nursing, 13(1), 10-14. doi:10.1016/j.ecns.2016.10.005
5
Gonzalez, L., & Sole, M. L. (2014). Urinary Catheterization Skills: One Simulated Checkoff Is Not Enough. Clinical Simulation in Nursing, 10(9), 455-460. doi:10.1016/j.ecns.2014.07.002
6
Hill, Kirsten, et al. “Psychomotor Skills for 21st Century: What Should Students Learn?” Curriculum Redesign, Center for Curriculum Redesign, Apr. 2018, curriculumredesign.org/wp-content/uploads/Psychomotor-Skills-CCR-May2018.pdf.
7
Howell, W. S. (1986). The Empathic Communicator. Prospect Heights, IL: Waveland Press.
8
Kemery, S, and Briyana Morrell. “Differences in Psychomotor Skills Teaching and Evaluation Practices in Undergraduate Nursing Programs.” Ovid, 19 June 2019, insights.ovid.com/crossref?an=00024776-900000000-99625.
9
Labbé, M., Young, M., Mascarella, M., Husein, M., Doyle, P. C., & Nguyen, L. H. (2020). How Consistent Is Competent? Examining Variance in Psychomotor Skills Assessment. Academic Medicine, 95(5), 771-776. doi:10.1097/acm.0000000000002985
10
Papaheorgiou, Konstantinos. Journal of Physical Education and Sport, 27 Feb. 2019, www.efsupit.ro/images/stories/April2019/Art92.pdf.
11
