Kathi R. Shepherd, RDH, MS Assistant Dean, Educational Development and Assessment Maureen VanHook, MSDH, Director, Dental Hygiene Program
ADEA Annual Session 2016
Faculty Calibration: An Essential Process for Shaping Future Competent Professionals
Increase knowledge of calibration strategies and the impact on preparing future competent professionals.
Implement engagement and acclimation strategies for full, part-time and graduate student faculty.
Design calibration best practices supporting and training full, part-time and graduate student faculty aimed at assessment of student
performance.
Seminar Objectives
Calibration concepts
Emerging technology
Pre and post training score analysis techniques
Learning Management Systems utilization
Faculty development programming
Systems that foster calibration process
Seminar Inclusions
Calibration Defined
Faculty implementing standardized teaching practices representing consistent application of theories and evaluation (Casa, ADHA Access, 2015)
Previous research has revealed inconsistencies in agreement among faculty members due to variations in clinical judgment (Partido et. al., 2015, JDE).
North American dental students identified inconsistent clinical feedback as one of the major obstacles in achieving clinical competence (Henzi et. al., 2006, JDE)
From the students’ perspective, clinical feedback has been perceived as more credible when the faculty were trained with exercises that
fostered subject mastery (Holyfield, Berry, JDE, 2008)
Impact of faculty variation on student performance is yet unknown (Lanning et. al., 2005, JDE)
Background
Inconsistent clinical evaluation among faculty has long been a source of frustration for students since lack of agreement among faculty can negatively affect students’ learning outcomes (Jacks et. al., 2008, JDE).
If grades are inaccurately or unfairly assigned, student morale may suffer, and motivation to improve skills may decline (Jacks et. al. 2008, JDE).
Lack of calibration among dental educators is “a significant source of trouble, worry, and discomfort; a major source of anger; and one of the primary reasons for abandonment of a quest for excellence and resignation to just getting by (Fuller, 1972, JDE).
Students believe that differences between their instructors affect their clinical progress.
Students may adopt “faculty-specific” strategies for addressing clinical problems instead of striving to meet the standard criteria (Lanning et. al., 2005, JDE).
Background
Calibration Defined
Faculty developed clinical evaluation systems for assessing student performance
Using criteria based student evaluation instruments to reproduce those standards in different situations (Haj-Ali, Feil, 2006, JDE)
Training that involves realistic situations and contexts
comparable to practice (Christie et. al., 2007, JDE, Drucker et. al. 2012, JDE)
Subsequent evaluation of the faculty in regards to implementing the system
Evaluation of the outcomes of the calibration program in regards to learner competence
Components of Well Designed
Calibration Programs
Student Achievement
of
Competency
Clinical Evaluation
System Developed By
Faculty
Training Involving
Criterion Referenced Evaluation Instrument Used
In Realistic Context
Subsequent Evaluation of Implemented Calibration
Program Assessment of
Learners‐Is Calibration Contributing to
Competency Achievement?
Cycle For Calibration
Factors to Consider
Previous experiences of faculty
Availability of all faculty for training
Mastery of educational principles
Acquaintance with clinical course objectives, procedure protocols
Bridging the gap between theory and practice
Knowledge of evidence to support procedure
Knowledge and application of formative and summative feedback principles
Differing faculty to student communication styles
Differing student abilities, professionalism
Complexity of patients
Generational differences
Creation of safe environment for discussion and calibration activities
Faculty “Buy In”
TIME!!!!!!!!!!!!!!!!!!!!
(Casa, ADHA Access, 2015, AAP Educator Workshop 2014)
Calibration Concepts
Identification of critical or unacceptable errors
Acceptable range of gold standard
Mastery approach
Error Identification Via the Use of Uploaded Student Videos and Criterion Referenced Process Evaluations
Faculty Standardization Session-Pre Demonstration of “Gold Standard”
Faculty asked to evaluate performance utilizing
criteria forms developed 1 year prior during annual assessment retreat‐
**majority of faculty present
Asked to evaluate
performance as if in the clinic
Error Identification Via the Use of Uploaded Student Videos and Criterion Referenced
Process Evaluation
Faculty very engaged, freely discussed opposition, concerns
Revised assessment criteria language
Revised instructions for assessment
Identified need for further sessions
Faculty Standardization Session-Observation
and Outcomes
Error Identification- Department of
Restorative Dentistry
Error Identification- Department of
Restorative Dentistry
Faculty Standardization Session 3 weeks later- Demonstration of “Gold Standard”
Presentation of “Gold Standard” videos
located on Bb course site
Provided live
demonstration via document camera
Blackboard Learning Management
System-Access for Students and Faculty
Faculty very engaged, freely discussed opposition, concerns
Identified differing techniques
Clarified “Gold Standard” technique in alignment with criteria form
Discussed differences between junior and senior student competency levels and
expectations/goals-CDCA preparation vs entering patient care
Faculty Standardization Session-Observation
and Outcomes
Conduct post demonstration
calibration session utilizing same student videos
Analyze pre and post evaluations
Share with faculty
Assess trends in Class of 2017 process evaluation/competency examination performance
Next Steps
Curriculum and Instruction Methods – Department of Restorative Dentistry Blackboard Website
Virtual Prosthodontics Website
Clinical Manual
HELP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Blackboard Learning Management
System-Access for Students and
Faculty
Lecture Capture- Techsmith Relay
PRIOR TO Assessment in Clinic:
1) Students select a patient from patient family
2) Complete an entire care plan and appointment plan for the patient based on actual axiUm patient record forms
3) Students present care plan by using document camera on the podium in the classroom as well as showing actual axiUm record forms via the
computer on the podium
4) Faculty provide assessment utilizing criterion based evaluation form used in clinic
5) Assessments analyzed by module leader/lecturer
6) Follow up session conducted with faculty to discuss trends and clarify areas in need of further explanation
7) At end of semester reassess care plan faculty assessment trends
Dental Hygiene Care Plan Faculty Standardization
Process via Student Seminar
144 DS 3 students, 288 patients
28 DS Faculty
1st clinic semester Oral Diagnosis/Treatment Planning
4 week program
HELP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Department of Biomedical and Diagnostic
Sciences “OD Bootcamp”
1. Each patient record reviewed by course director
2. Course director drafts treatment plan to communicate expectation of course director
3. Patients assigned to students based on degree of difficulty
4. Faculty assigned to 2 students based on faculty expertise and student competency level
5. Student Treatment Plan reviewed by course director
6. Feedback provided to student and supervising faculty member
Department of Biomedical and Diagnostic
Sciences “OD Bootcamp”
Mandates faculty to score within an acceptable range of the gold standard prior to providing clinical instruction- each instructor should be required to achieve a certain level of agreement with a gold standard in order to evaluate student work
Instructors should become “competent” evaluators
Serves as a standard of rater reliability in order to promote competent instructors that provided consistent clinical
instruction.
The implication is that a rater who cannot attain that level of competence would be expected to continue working on improving this skill perhaps under the direct mentoring of the individual who represents the gold standard. The cycle of training, testing, mentoring, and further testing would
continue until that agreed-upon level of mastery had been reached.
Mastery Approach to Calibration (Mackenzie, 1973, JDE)
Faculty Standardization: Clinic Orientation
All faculty and students attend a
clinic orientation at the beginning of each new term.
All process evaluations and competency evaluations are distributed and reviewed.
All clinic protocols and policies are reviewed.
Restorative Dentistry DS 3 Clinical
Orientation
Prescription Evaluation
Ongoing Need for Calibration: New CODA Standards i.e. Laboratory Prescription
Communication Competency
Faculty Calibration: A Guide to Success, American Academy of Periodontology, 2014
Other Attempts at Standardization-UDM Department of Periodontology and Dental Hygiene Bb Faculty Discussion Board
Standardization via the Use of Blogs
Techniques for Creating a Safe Environment- Use of Clickers
Taken from: 2014 AAP Educator Calibration Workshop. Presentation given by P. Luepke, DDS, MS Marquette University School of Dentistry
Techniques for Creating a Safe Environment- Use of Other Anonymous Tools
Professional Faculty Development
Anecdotal Observations
review of studentsʹ course
evaluations, external curriculum
review
faculty development &
calibration programs
improvement in consistency of
faculty assessment of
student performance
What makes this process successful?
Administration support
Allowing adequate time during each month- ½ day departmental academic planning session/month
Allotting funds for faculty development- $1,500.00 annually for travel, additional online AAL funding
Faculty that are willing to attend, engage and be open to change
Course directors willing to conduct sessions and provide continual standardization activities
Chairpersons facilitating the process
Who Facilitates Calibration Sessions?
Content Experts ( Course Directors)
Assistant Deans (policies, protocols, systems)
Program Directors
Clinic Coordinators/Course Directors
New Faculty Educational Development Plan
Roles and Responsibilities of Course Directors, Faculty documents
Chairperson Job Description
Rank and Tenure process/review
Professional Development Planning Process
Promoting Faculty Engagement
IRB approval submission for Faculty Calibration program
Pre test Qualtrics Survey
Calibration program utilizing cases with clickers for periodontology and dental hygiene faculty
Post test Qualtrics Survey
Analyze trends
Analyze faculty assessment trends
Analyze student competency
Next Steps- Department of
Periodontology and Dental Hygiene
Computerized assessment tool in
CadCam- prep scanned, student visualizes prep-helps faculty standardization due to computer accuracy (IADR poster
comparing faculty vs computer feedback)
Structured calibration online modules that must be completed (including assessment) before teaching in simlab, clinic
Next Steps- Department of Restorative
Dentistry
Facilitate safe environment
Spread the wealth with multiple people leading sessions
Use multiple methods to accommodate all faculty needs
Imbed calibration session time on academic calendar during day
Record sessions for those not able to participate face to face
Suggestions
Suggestions
Dr. Nahid Kashani, Chairperson,
Department of Biomedical and Diagnostic Sciences
Dr. Dincer Gurun, Chairperson,
Department of Restorative Dentistry
Dr. Tameka Sloan- Thompson, Director, Predoctoral Clinical Periodontics
