DEVELOPMENT AND ASSESSMENT OF DISCRIMINATION EXCERCISES FOR FACULTY CALIBRATION IN PRECLINICAL OPERATIVE
DENTISTRY
SUMITHA NAZAR AHMED
A thesis submitted to the Faculty of The University of North Carolina at Chapel Hill in partial fulfillment of the r equirements for the degree of Master of Science in
the Department of Operative Dentistry at the School of Dentistry
Chapel Hill 2014
ii © 2014
ABSTRACT
Sumitha Nazar Ahmed: Development and Assessment of Discrimination Exercises for Faculty calibration in Preclinical Operative Dentistry
(Under the direction of Lee Boushell)
To my husband Yashin Ahmed, my children Adi and Ian, your love and support made this dream come true .
To my parents Naziruddin and Jameela Nazar who taught me that success come from following one’s heart
ACKNOWLEDGEMENTS
To my Thesis Committee members, Dr. Lee Boushell, Dr. John Sturdevant, Ms. Rebecca Wilder and Dr. Vicki Kowlowitz, I am grateful for your time,
encouragement and guidance
Dr. Terry Donovan, Dr. Harald Heymann, Dr. Andre Ritter and Dr. Edward Swift, it has been an honor to be your student
To the staff of the Department of Operative Den tistry, Shannon Tate, Dayna McNaught, Jamie Desoto, Rosanna Arrington, Barbara Walton and Cynthia
Lambert. Thank you for always being there, ready to help .
I would like to thank Dr. Rick Walter and Dr. Leinfelder for taking the time to participate in my thesis project. Dr. May, Dr. Eidson and Dr. Zandona, thank you
for making me feel like a part of the Operative family.
Dr. Ceib Phillips and Ms. Debbie Price, I thank you for your time and guidance with the statistical analysis. Dr. Wilder thank you for being so patient with me and
all the help you have given me over the years.
Alex and Silvia you have been the best classmates. I will never forget these three years of sharing accomplishments, success and failures. You will be in my heart
forever
To all the Operative residents who made this journey memorable, Ayesha, Roopsie, Upoma, Villi, Clay, Tan, Mohammad, Anmar and Taiseer. Thank you!
vi
TABLE OF CONTENTS
LIST OF TABLES ... viii
LIST OF FIGURES ... ix
LIST OF ABBREVIATIONS ... xi
INTRODUCTION ...12
REVIEW OF LITERATURE ...14
LIST OF REFRENCES ...24
MANUSCRIPT ...27
1. INTRODUCTION ...27
2. SPECIFIC AIMS ...30
3. MATERIALS AND METHODS ...31
3.1 DENTIFORM TEETH...31
3.2 PANEL OF EXAMINERS ...31
3.3 SELECTION OF DENTIFORM TEETH ...32
3.4 CALIBRATION SESSION ...32
3.5 DISCRIMINATION EXERCISE RATIONALE ...33
3.6 DISCRIMINATION EXERCISE DESIGN ...33
3.6.1. PROXIMAL CONTACT CLEARANCE ...33
3.6.3. RETENTION GROOVE DEPTH...36
3.6.4. PREPARATION FINISH ...36
3.6.5. PREPARATION MARGIN ...37
3.7 REVIEW OF ASSESSMENT RUBRIC ...38
3.8 TIMING OF PHASE I AND PHASE II ASSESSMENT ...39
3.8.1. PHASE I ASSESSMENT ...39
3.8.2. PHASE II ASSESSMENT ...39
3.9 ORGANIZATION OF THE DATA ...39
4. STATISTICAL ANALYSIS ...39
5. RESULTS ...40
5.1 INTEREXAMINER RELIABILITY – PHASE I ...40
5.2 INTRAEXAMINER RELIABILITY – PILOT STUDY TO PHASE I ...41
5.3 INTEREXAMINER RELIABILITY – PHASE II ...41
5.4 INTRAEXAMINER RELIABILITY - PHASE II ...42
6. DISCUSSION ...43
7. CONCLUSIONS...44
viii
LIST OF TABLES
TABLE 1 - Class II Amalgam Procedure Performance Assessment Rubric
Used For the Pilot Study, Phase I and Phase II...49 TABLE 2 - Average percent agreement (mean +/- sd) and 95% CI for
assessment of components of Class II cavity preparation during the Pilot study ...51 TABLE 3 - Class II amalgam procedure assessment rubric that was used
to define the level of student performance and associated point values for each component ...69 TABLE 4 - Average percent agreement (mean +/- sd) and 95% CI for
assessment of components of Class II cavity preparation during Phase I ...71 TABLE 5 - Average percentage intra examiner agreement for
each examiner when comparing the results from Pilot Study to Phase I ...73 TABLE 6 - The level of intraexaminer variation (%) for all 13 components
of Class II cavity preparation for each examiner when comparing Pilot study to Phase I ...76 TABLE 7 - Average percent agreement (mean +/- sd) and 95% CI for
assessment of components of Class II cavity preparation during Phase II ...78 TABLE 8 - Average percentage intraexaminer agreement for each examiner
when comparing the results from Pilot Study to Phase II ...79 TABLE 9 - Average percentage intraexaminer agreement for each
examiner when comparing the results from Phase I to Phase II ...80 TABLE 10 - The level of intraexaminer variation (%) for all 13
components of Class II cavity preparation for each examiner when comparing the
Pilot Study to Phase II...83 TABLE 11 - The level of intraexaminer variation (%) for all 13
ix
LIST OF FIGURES
FIGURE 1 - Average percentage agreement among examiners after
assessment of 13 procedural components of Class II cavity preparations, identified in the pilot study...50 FIGURE 2.1- The mean diameter of the UNC 15 probe at the 4-5 mm
mark was 0.5 +/-0.02 and 0.75 +/-0.02 at the 11-12 mm mark. The measurements were made using the Image J ® software ...52 FIGURE 2.2- 3D model used to demonstrate clinically acceptable
example of proximal contact clearance...53 FIGURE 2.3 - 3D model used to demonstrate clinically unacceptable
example of proximal contact clearance...54 FIGURE 3.1- 3D model demonstrating clinically unacceptable retention
groove placement in the enamel facial wall of a preparation with ideal axial wall depth ...55 FIGURE 3.2- 3D model demonstrating clinically unacceptable retention
groove placement at the DEJ of the facial wall of a preparation with ideal axial wall depth ...56 FIGURE 3.3 - 3D model demonstrating clinically acceptable retention
groove placement ~0.2 mm internal to the DEJ such that it is partially in the dentin facial wall and partially in the adjacent axial wall of a preparation with ideal axial wall depth ...57 FIGURES 3.4- 3D model demonstrating clinically unacceptable retention
groove placement in the axial wall of a preparation with ideal axial wall depth...58 FIGURES 3.5- 3D model demonstrating clinically unacceptable retention
groove placement in the gingival wall of a preparation with ideal axial wall depth ...59 FIGURES 3.6- 3D model demonstrating clinically acceptable retention
groove placement ~0.2 mm internal to the DEJ in the facial dentin wall of a preparation with deep axial wall depth ...60 FIGURES 3.7- 3D model demonstrating clinically unacceptable retention
groove placement in the line angle of the facial and axial walls of a preparation with deep axial wall depth ...61 FIGURES 4.1 & 4.2 – 3D model demonstrating clinically acceptable
x
FIGURES 4.3 & 4.4 – 3D model demonstrating clinically unacceptable
preparation finish of smooth walls and abrupt transitions ...63 FIGURES 4.5 & 4.6 – 3D model demonstrating clinically unacceptable
preparation finish of rough walls and gentle transitions ...64 FIGURES 4.7 & 4.8 – 3D model demonstrating clinically unacceptable
preparation finish of rough walls and abrupt transitions ...65 FIGURE 5.1- Image of the 3D model with a superimposed protractor
that demonstrates clinically unacceptable preparation wall orientation of < 800 ... 66 FIGURE 5.2- Image of the 3D model with a superimposed protractor
that demonstrates clinically unacceptable preparation wall orientation of >1000 ... 67 FIGURE 5.3- Image of the 3D model with a superimposed protractor
that demonstrates clinically acceptable preparation wall orientation of approximately 900 ... 68 FIGURE 6- Average percentage agreement after assessment of 13
procedural components of Class II cavity preparations among examiners at Pilot Study and Phase I ...71 FIGURE 7 - The component ‘Enamel Present’ had the lowest intraexaminer
variation from Pilot Study to Phase I ...74 FIGURE 8 - The component ‘Proximal Contact Clearance’ had the
highest intraexaminer variation from Pilot Study to Phase I ...75 FIGURE 9- Average percentage agreement among examiners after
assessment of 13 procedural components of Class II cavity preparations,
identified in the post...77 FIGURE 10 - The component ‘Isthmus Width’ had the lowest intraexaminer
variation from Pilot Study: Phase I: Phase II ...81 FIGURE 11 - The component ‘Proximal Contact Clearance’ had the highest
xi
LIST OF ABBREVIATIONS
3D 3 Dimensional CI Confidence Interval DEJ Dentino-enamel Junction EM Electromagnetic
ICDAS International Caries Detection and Assessment System IRB Institutional Review Board
MOD Mesial Occlusal Distal SD Standard Deviation
UNC University of North Carolina at Chapel Hill US
GG CL CLSC
The United States of America Glance and Grade
Checklist
12 Introdu cti on
Dental st udents under go ri gorous pre -cli nical t raining in preparation for ent rance int o the cli nical set ting. Dental facult y utiliz es s imul at ed cli nical setti ngs in pre-clini cal cours es for the purpose of devel oping and ass essi ng dental students’ knowledge and skills pre-requisite for patient care.
Cons ervative Operati ve Denti str y (DENT112) is an exampl e of a pre -cli ni cal operative dentist r y t aught at t he UNC S chool of Dentist r y.
In t he di dacti c session of the pre-clini cal operative course the students learn the theor y behi nd di fferent desi gns of cavit y preparati ons . During the hands -on (or laborat or y) session of t he course, the y us e this t heoreti cal inform ation t o devel op the hand skil l necessar y for preparing and rest oring caviti es on denti form teeth that are mounted in maniki ns. As part of the pre -clini cal course, the y com e i n contact wit h multiple facult y who grade thei r work. Learning Theor y suggests t hat it is ver y i mport ant that t he informat ion gi ven t o the students rem ain consist ent from one inst ruct or to the other in order to avoi d an y confusi on among t he students.
As fut ure dentist s, dent al students are ex pect ed t o m ake cli ni cal
13
are consis tent with each other as the y communi cate concepts to thei r st udent s. Act ive st eps must be taken to increas e t he l evels of consist enc y, or agreem ent or calibrat ion am ong facult y. Poor consist enc y among facul t y m a y lead to
14
Review of Literatu re
Evaluation m ethods can be broadl y clas si fied i nto subj ective eval uati on and objecti ve evaluation. Whil e the word subj ective can change from pers on to person, obj ective m eans a fix ed real it y. S ubj ecti vit y associ at ed wit h gradi ng student perform ance can l ead t o confusion and stres s among t he students. Full er stat es that, in an evaluati on s yst em gui ded b y subjecti vit y, it i s often diffi cult for the faculty to “defend” his/her score to the student’s satisfaction or his/her own.1
Accordi ng to J enkins et al , the p robl ems with facult y cons ist enc y m a y lead student s t o perceive t hat evaluation meth ods are s omewhat arbit rar y. The y suggest ed t hat this concept can undermine t he learni ng proces s and produce a negati ve effect on undergradu at e’s confidence and perform ance.2 Their fi ndi ngs support the i dea that consist enc y in the feedback provi ded to t he students hel ps to improve thei r perform ance and also avoids am bi gui t y as sociat ed with grading procedures. Mackenzie recommends ‘maximizing diagnostic feedback’ (using obj ecti ve evaluation methods ), for the purpose of student l earning.3 This can onl y be accom plis hed b y est ablishing a s ys tem th at makes use of obj ective evaluati on m et hods .
The cons ist enc y or degree of agreem ent between facult y m embers,
15
and int erexaminer reliabi lit y. Int raexami ner reliabilit y des cri bes the consist enc y of a single examiner in grading the same sample on multi pl e occas ions .4
Int erexami ner reli abi lit y m easures t he degree of agreem ent am ong t he examiners when t he y evaluat e t he perform ance of the sam e group of st udents on the sam e task.5 Studies in the field of facult y cal ibrat ion have s hown t hat est abl ishi ng agreem ent among facult y m em bers is not an eas y t ask. This can be due to the inconsis tent gradi ng methods , di ffering rating s cal es, individual t eaching philosoph y and s o on.
Clinical perform ance can be judged i n t erms of it s 3 pos sibl e out com es. Perform ance that: 1) is clini call y accept able, 2) needs improvement and 3) is clini call y unaccept able. Am erican dent al schools have hi stori call y ass ess ed perform ance i n pre -clinical simulat ions of the cli ni cal envi ronment b y usi ng 3 methods: 1) Glance and Grade (GG), 2) C heckli st (C L) and 3) Checkli st with Specific Cri teri a (C LSC). The Gl ance and Grade method ut ilizes a subj ective gl obal as sessm ent of the st udent perform ance without s peci fi c evaluati on of each component of t he skill . of a skill .6 The Checklis t an d C heckli st with Specific Cri teri a m et hods s eek to util ize a more anal yt i cal form of evaluation where each com ponent of t he enti re performance is eval uat ed separatel y on a writ ten checklist . The Checkli st with Specific Cri teri a m ethod als o defines speci fi c l evels of performance of each component .
Houpt and Kress suggest s that global ass essm ent can be us ed for the purpose of certi ficati on, i .e., in a situati on where the student perform ance needs to be evaluated as ‘satisfactory’ or ‘unsatisfactory’.7
16
assessment as a ‘product approach' and points out that it should be
suppl em ent ed wi th other forms of evaluat ion. He argues that l ooking onl y at the measure of t he end - product, wit hout anal yzi ng t he factors that contri but e t o t he vari ations in quali t y of t he product, resul t s in wast eful repetiti on of t he
evaluati on process .3
Bot h global and anal yt i cal approaches have been t est ed to det erm ine the level of int ra- and i nter exam iner reli abil i t y. Goepferd and Kerber8 developed and t est ed t he effi ci enc y of an anal yt i cal s yst em for evaluati ng clas s II cavit y preparati on on pri mar y t eeth. The int ent of the anal yt i cal s ys t em was t o decreas e the subjecti vit y of cl inical eval uation and to i ntroduce obj ect i ve meas ures in order to increas e examiner reli abilit y. Thi s s ys tem was then compared with the traditi onal gl ance and grade m ethod t o determine degree of i nt ra - and i nter - examiner reliabilit y with bot h m ethods . The stud y found that t he overall interexaminer rel iabi lit y im proved with the anal yt ical s yst em; however t he findings we re not t es ted for st atisti cal si gnifi cance.
Addit ional res earch com paring GG, C L and C LSC m ethods was accomplis hed at t he Universit y of North Carolina i n 1983. The i ntra - and
interexaminer rel iabi lit y in precli ni cal pedodonti c grading was eval uated. It wa s concl uded t hat no m ethod yi el ded superi or i nt ra - or int erexaminer reli abilit y.9
17
rat ing s cal es : a t wo -point s cal e wit h t wo speci fi ed points, a fi ve -poi nt scal e with end points specifi ed an d fi ve-point s cale with all end poi nts defi ned. The
findings of t he stud y demons trat ed that when crit eri a were defined speci ficall y, examiners t ended t o be more accurate in t hei r j udgm ents. However, the use of a two -poi nt rating s cal e was found t o have mo re i nterexaminer agreement t han the use of a five-poi nt rating scal e even though a fi ve -point s cale ma y be more benefi cial for instructional purpos es.7 Si milar results b y Hinkelm an and Long showed sli ghtl y more i nterexaminer agreem ent i n a t wo -point eval u at ion s ys tem (pass - fail) than a three -point scali ng s ys tem (‘no improvem ent necess ar y’, ‘clinically acceptable’ and ‘clinically unacceptable and uncorrectable’).
However, the aut hors suggest t hat the three -point evaluation s ys tem was more useful for ranki ng according t o abilit y.1 0
Although the resul ts of t hese st udi es i ndi cate that a t wo -point scal e l eads to great er examiner reli abilit y, the y hi ghl ight t hat it gives littl e t o no
18
Hinkelm an and Long stud y who found increas ed int erexaminer reli abilit y i n thei r col laps ed t wo -point scori ng s yst em over t hree -poi nt s cal e. The exami ners who lik ed the t wo -option scori ng were t hose with l ess experi ence. The y disli ked the range of compet enc y ass essm ent options and the l evels assi gned t o some cri teri a. Thos e who preferred the t hree -option s coring m ethod report ed it
allowed t hem to recognize superior work and, at t he s am e tim e, avoi d unrealisti c expect ations of perfection.
Research in t he fiel d of facult y calibrat ion has als o s ought t o anal yz e speci fi c cri teri a, us ed in as sessm ent methods, in terms of clari t y of t he verbiage. Studi es t hat have carefu ll y defi ned checkpoint s, des i gned to decreas e ambi guit y, have obt ained fai rl y hi gh agreement among the exam iners.1 2 As defined b y
Mackenzi e et al, eli minat ing probl ems with the eval uation process includes, but are not limit ed to, it ems such as unst andardize d aids to eval uat e, i ncompl et e operational definiti ons, dis crepanci es in visual acuit y and inadequac y in verbal definit ions . The y concl uded t hat the impact of t hese probl em s can be reduced b y using checklists wi th definitions t hat des cribe how to make obse rvations and how to cat egoriz e the obs ervations with minim al inference or subj ective bi as.1 3
Sharaf et al , has s uggest ed that the goal of educat ors shoul d be to desi gn a m ethod of ass essm ent that i s both obj ective and reli abl e s o as to reduce
fri cti on between s tudents and facult y over the iss ue of gradi ng.1 4 In an att em pt to m ak e the eval uati on process more objective , Schiff et al desi gned a devi ce called the “pulpal floor measuring instrument” to measure the profile of
19
aut hors report ed si gnifi cant i m provem ent in operator co nsist enc y using t his instrum ent.1 5 C ardos o et al used the Kavo PrepAssis tant® t o objectivel y
evaluat e s tudent to ot h preparations. The machine was us ed t o com pare an i deal tooth preparation b y the facult y m ember agai nst a preparation com plet ed b y a student . Bot h t he preparations were di git all y s canned and a data s heet
contai ning preparati on anal ys is and feedback was t hen proces sed which was gi ven t o the student. The authors found t he use of the machine in its full pot enti al b y t he preclinical depart ment t o be ver y l abor int ens ive. Even with a greater guarant ee of obj ecti ve evaluation, the m achine was not able to as se ss all the components of the t ooth preparation, l eaving 30 % of t he eval uati on to be com plet ed b y the facult y. 1 6
20
Such devi ces hel p t o ensure that t he st udent recei ves an obj ective evaluation of thei r work in addition to consis tent feedback . It should be not ed that thes e s yst ems currentl y cannot be us ed to provi de form ative and s ummati ve feedback for st udent perform ance of dent al restorations.
In t he abs ence of such devi ces t he course director rel ies on the preclini cal cours e facult y to prov ide consist ent sum mative and form ative feedback.
Att empts to increas e facult y consist enc y t hrough improved communi cation of speci fi c perform ance crit eri a, rati ng scal es, and/ or t rai ning have met wit h
inconsis tent result s .2 3 One st ud y found that facult y meeti n gs desi gned t o resolve inconsis tencies di d not re s ult i n i ncreased calibration.2 4
Nat kin and Guil d report ed a st atisti call y s igni ficant increas e in
interexaminer rel iabi lit y after t he trai ning sessions. Each s es si on consist ed of the exam ination of t en randoml y s el ected student proj ects foll owed b y
discussi on of the crit eri a establis hed for t hos e proj ects .2 5 A lit erat ure revi ew b y Patridge and M ast f ound t hat st udi es on facult y trai ning yi elded inconsist ent results .2 6 Som etim es, the inconsist enc y and l ack of fair grading is due t o the abs ence of cl ear guidelines on how t he ex aminers should eval uat e clini cal perform ances .2 7
21
intraexaminer rel iabi lit y parti cul arl y for t he most experi enced examiner.2 8 Philip Feil recomm ends t he us e of two examiners gradi ng independentl y but working in associ ati on with one another to develop a biparti s an approach, whi ch will t heoreti call y facilitat e a more uni form met hod of ass es sm ent b y reduci ng the e ffect of individual bi as.2 9
Scruggs et al conducted a pilot st ud y to i nvesti gat e t he us e of speci fi c cri teri a and facul t y calibration on the reli abili t y of inexperi enced examiners on dental s ealant evaluations . The obj ectives of the s tud y were to identi f y
differences i n calibrat ed and non -cali brat ed exami ners . The examiners were calibrat ed b y an expert and a non -expert to ass ess reli abil it y among the st ud y parti cipant s. The y us ed an anal yti cal, crit eri a bas ed evaluati on s yst em. The results showed that t he calibrat ed group had a hi gher reli abilit y than the non -calibrat ed group. One im portant finding of the stud y was t hat t he group calibrat ed b y the expert act uall y decreased in int erexami ner reliabi lit y aft er training. Based on t hes e resul ts, the y concl uded t hat the met hod us ed for calibrat ing examiners ma y be of m ore i m port ance t han t he ex peri ence of the calibrat or when planning a training s essi on.3 0
22
foll owi ng t raini ng, and t en weeks lat er. The y concluded t hat , with training, interexaminer rel iabi lit y with a gol d st andard can be improve d and such improvement i s reas onabl y resi stant t o det eri orati on after ten weeks.2 3
Salvend y et al , in their arti cl e from 1973 , comm ent ed that “A dent al student can acqui re his ski lls m ost effecti vel y when he is provided with and evaluat ed on obj ecti ve and qualit ative criteria m easures”.3 1 The c rit eri on
ori ent ed m ethod, b y definit ion, reduces t he subj ectivit y as sociat ed with grading. If the descripti on of the crit eria leaves no room for s ubj ective interpretation, then it onl y enhances the communi cation bet ween the student and the exami ner.
Review of salient li t erat ure t o dat e l ends support to the notion that assess ment m ethods that utiliz e C LSC afford the bes t possi bili t y of achi evi ng reas onabl e l evel s of facult y calibration. However, us e of C LSC al one does not ens ure calibration among course fac ult y. Met hods of communicating the
meaning and appli cat ion of each crit eri on must be identi fi ed. Incorporati on of standardized m easurem ent i nst rum ent s where ever pos sibl e becom es es senti al. The Departm ent of Operative Dentist r y at the Universit y of North C arolina uses C LSC to support t he teaching of princi pl es inherent t o t he dis ciplines of
23
24
REFERENCES
1. Fuller JL. The effects of training and criterion models on interjudge reliability. Journal of dental education. Apr 1972;36(4):19-22.
2. Jenkins SM, Dummer PM, Gilmour AS, Edmunds DH, Hicks R, Ash P. Evaluating undergraduate preclinical operative skill; use of a glance and grade marking system. Journal of dentistry. Nov 1998;26(8):679-684.
3. Mackenzie RS. Defining clinical competence in terms of quality, quantity, and need for performance criteria. Journal of dental education. Sep 1973;37(9):37-44.
4. Dhuru VB, Rypel TS, Johnston WM. Criterion-oriented grading system for preclinical operative dentistry laboratory course. Journal of dental education. Sep 1978;42(9):528-531.
5. Brown G, Manogue M, Martin M. The validity and reliability of an OSCE in dentistry. European journal of dental education : official journal of the Association for Dental Education in Europe. Aug 1999;3(3):117-125.
6. Vanek HG. Objective evaluation of dental student technic products. Journal of dental education. Mar 1969;33(1):140-144.
7. Houpt MI, Kress G. Accuracy of measurement of clinical performance in dentistry. Journal of dental education. Jul 1973;37(7):34-46.
8. Goepferd SJ, Kerber PE. A comparison of two methods for evaluating primary class II cavity preparations. Journal of dental education. Sep 1980;44(9):537-542.
9. Vann WF, Machen JB, Hounshell PB. Effects of criteria and checklists on reliability in preclinical evaluation. Journal of dental education. Oct 1983;47(10):671-675.
10. Hinkelman KW, Long NK. Method for decreasing subjective evaluation in preclinical restorative dentistry. Journal of dental education. Sep 1973;37(9):13-18.
11. Deranleau NJ, Feiker JH, Beck M. Effect of percentage cut-off scores and scale point variation on preclinical project evaluation. Journal of dental education. Oct
1983;47(10):650-655.
12. Ryge G, Snyder M. Evaluating the clinical quality of restorations. Journal of the American Dental Association. Aug 1973;87(2):369-377.
25
14. Sharaf AA, AbdelAziz AM, El Meligy OA. Intra- and inter-examiner variability in evaluating preclinical pediatric dentistry operative procedures. Journal of dental education. Apr 2007;71(4):540-544.
15. Schiff AJ, Salvendy G, Root CM, Ferguson GW, Cunningham PR. Objective evaluation of quality in cavity preparations. Journal of dental education. Feb 1975;39(2):92-96. 16. Cardoso JA, Barbosa C, Fernandes S, Silva CL, Pinho A. Reducing subjectivity in the
evaluation of pre-clinical dental preparations for fixed prosthodontics using the Kavo PrepAssistant. European journal of dental education : official journal of the Association for Dental Education in Europe. Aug 2006;10(3):149-156.
17. Long AF, Mercer PE, Stephens CD, Grigg P. The evaluation of three computer-assisted learning packages for general dental practitioners. British dental journal. Dec 10-24 1994;177(11-12):410-415.
18. Pollard DJ, Davenport JC. An evaluation of training general dental practitioners in partial denture design using a computer-assisted learning program. British dental journal. Dec 10-24 1994;177(11-12):405-409.
19. Grigg P, Stephens CD. Computer-assisted learning in dentistry. A view from the UK. Journal of dentistry. Jul-Aug 1998;26(5-6):387-395.
20. Buchanan JA. Experience with virtual reality-based technology in teaching restorative dental procedures. Journal of dental education. Dec 2004;68(12):1258-1265.
21. LeBlanc VR, Urbankova A, Hadavi F, Lichtenthal RM. A preliminary study in using virtual reality to train dental students. Journal of dental education. Mar 2004;68(3):378-383.
22. Welk A, Maggio MP, Simon JF, et al. Computer-assisted learning and simulation lab with 40 DentSim units. International journal of computerized dentistry. 2008;11(1):17-40.
23. Haj-Ali R, Feil P. Rater reliability: short- and long-term effects of calibration training. Journal of dental education. Apr 2006;70(4):428-433.
24. Robertello FJ, Pink FE. The effect of a training program on the reliability of examiners evaluating amalgam restorations. Operative dentistry. Mar-Apr 1997;22(2):57-65. 25. Natkin E, Guild RE. Evaluation of preclinical laboratory performance: a systematic
study. Journal of dental education. Jun 1967;31(2):152-161.
26. Patridge MI, Mast TA. Dental clinical evaluation: a review of the research. Journal of dental education. Jun 1978;42(6):300-305.
26
European journal of dental education : official journal of the Association for Dental Education in Europe. May 2010;14(2):92-98.
28. Lilley JD, ten Bruggen Cate HJ, Holloway PJ, Holt JK, Start KB. Reliability of practical tests in operative dentistry. British dental journal. Sep 3 1968;125(5):194-197.
29. Feil PH. An analysis of the reliability of a laboratory evaluation system. Journal of dental education. Aug 1982;46(8):489-494.
30. Scruggs RR, Daniel SJ, Larkin A, Stoltz RF. Effects of specific criteria and calibration on examiner reliability. Journal of dental hygiene : JDH / American Dental Hygienists' Association. Mar-Apr 1989;63(3):125-129.
27
MANUS CRI PT 1. In trodu cti on
Operati ve denti str y c oncepts and t echni ques are init ial l y i ntroduced to first year dent al students through parti ci pation in a precli ni cal operati ve dentist r y course. The course cont ains a di dacti c portion during whi ch students learn the theoret ical a spects of operative dentist r y. The students rel y on
multipl e facult y for applicati on, reinforcem ent and enhancem ent of theoreti cal princi ples duri ng a s imult aneous l aborator y portion of t he cours e. Facult y members t eaching the cours e are expect ed to provi d e consi stent formati ve and summ ative feedback of t he st udent performance. Low agreem ent, with regard t o assess ment of student perform ance, among facult y m a y lead to student confusion and frustrati on. Att empts t o increas e facult y agreem ent through improved
comm uni cation of s pecifi c perform ance crit eri a, rati ng scal es, and/or t raini ng have met wi th i ncons istent results .1 Few studi es have carefull y anal yzed each of t he components of the evaluation s yst em being us ed so as t o identi f y speci fic areas of low agreem ent and then t aken t arget ed st eps , through facult y
calibrat ion trai ning, to improve agreem ent.2
28
level of agreem e nt among the examiners when t he y are exam i ning the perform ance of the s am e group of student s on the s am e t as k.4 Int raexami ner rel iabilit y des cribes the consi stenc y of a singl e exami ner i n gradi ng t he s am e sample on multi pl e occas ions .3 Studies in the fiel d of facult y calib rat ion have shown t hat est ablis hi ng agreem ent among facult y m em bers is difficult. Thi s m a y be due t o the inconsi stent grading m ethods, di ffering rating s cal es and
indivi dual t eaching philos oph y5 - 9.
Operati ve Denti str y procedures accompl i shed in the cli nical s etting are generall y as sess ed as eit her cli nicall y accept abl e or clini call y unaccept abl e. However, the pre -cli nical t raini ng of thes e procedures requires that t h e y be further subdivided i nto indivi dual com ponents/st eps so as to ai d the s tudent duri ng the l earning process. Thi s allows assessm ent of concept ual underst anding as well as the abili t y to im pl ement each component of t he procedure. Overall procedural com pe tence m a y t hen be assessed as a net sum of all indi vidual com ponent s.
29
It is general l y accept ed that l evels of agreem ent should minim all y exceed that which woul d happen b y chance (50%) alone. Therefore, for t he purpos e of this s tud y, 60% was arbit raril y set as the minimum level of agreem ent an d average percent age agreement t hat fell below 60% was considered poor. The average percent age agreement (m ean and standard devi ati on with confidence intervals ) for each of the 13 components asses sed during the Pi lot st ud y are listed in Tabl e 2.
Anal ysi s o f t he ass es sments from t he pil ot stud y reveal ed six out of
thirt een cavit y preparat ion components had int erexami ner reli abili t y bel ow 60% (Fi gure 1). The six components of Class II cavit y preparation t hat needed
further facult y calibration were as foll ows: 1) P roximal and Gingival C ont act Clearance, 2) Retenti on Groove Pl acement, 3) R et enti on Groove Depth, 4)
Preparation M argin Finish, 5) P reparation M argin Orient ation and 6) P reparati on Toil et. Thes e various preparation features are des cribed i n furt her de t ail in mat eri al s and methods.
Evaluation of t he pil ot st ud y suggest ed l evels of i nherent facult y
30 2. Sp ecifi c Ai ms :
1. To i denti f y t he l evel of int erexam iner rel iabilit y among pre -clini cal operative dentist r y facult y when ass essing 32 (t hi rt y t wo) Cl ass II preparations perform ed b y 1s t yea r dent al students.
2. To develop t arg et ed exercis es desi gned to enhance the abi l it y of facult y t o dis crim inate among l evel s of st udent performance (discrimi nation exerci ses ) of various procedural components where average agreem ent among facult y was found to be below 60% and t o organize and pr es ent thes e
discrimi nation ex erci ses to i ndi vidual facult y m em bers as part of a cali brati on sess ion.
3. To eval uate the abilit y of di scrimi nat i on exerci ses (as reveal ed b y inter - and int raexam i ner reliabilit y) to i ncreas e initi al level s of facult y
calibrat ion in as sess ment of the components of C las s II preparation perform ed b y first year dent al s tudents accom plished during t hei r pre -cli nical operati ve dentist r y course (referred to as P hase I).
31 3. Materi als and methods
This was a longitudi nal , non -randomiz ed cohort st ud y conduct ed during the peri od of 2011 – 2013 at the Uni versi t y of North C aroli na School o f
Dentist r y. This stud y was IR B exempt (#12 -0262) b y t he Universit y of Nort h Carolina’s Institutional Review Board.
3.1 Dentiform teeth
According to the Webster’s dictionary, the word ‘Dentiform’ is defined as having the s hape of t ooth or t eet h. The dent iform teet h used for the current stud y were a model of tooth #30 with MOD cari es, model # A27A -46U, Kil gore® Internati onal Inc. The denti form teeth had been fabricat ed wi th a com posit e mat eri al crown, epox y resin dentin contained simul at ed dentin caries. The enamel and dent in anatom y was morphologicall y sim ilar t o a natural
mandibul ar fi rst m ol ar and the pl acement of the s imul at ed caries was simil ar to that found in the Cl ass II caries di sease s tat e.
3.2 Pan el of Examin ers
32 3.3 S el ecti on of d en tiform teeth
Thi rt y t wo Cl ass II preparations [ repres enting i deal (n = 8), accept abl e (n = 8), correctabl e (n = 8) and unaccept abl e (n = 8) student performance] were random l y sel ect ed from a pool of 82. Th e sam e 32 Class II preparations were assess ed b y the exam iners as part of t he P ilot, P hase I and P hase II parts of the stud y. The cavit y preparations were complet ed b y fi rst year dental students as part of the pre-cli nical operati ve dentistry cours e. The s tudents were ins truct ed to prepare an ideal (according to specifi c crit eri a) M OD cavit y preparation for am al gam restoration, with compl et e removal of the simul at ed cari es lesion. The preparati ons were accom plished in the si mulat ion laborator y desi gned to
repli cat e the clini cal set ting. The denti form toot h #30 was placed in t he dentiform wi th adj acent teet h (toot h # 29 and # 31) forming proximal contact point s, during the preparati on. The crit eria for cavit y preparat ion were adopt ed from Sturdevant’s Art and Science of Operative Dentistry, 5t h
Edition.1 0 3.4 Calib ration Ses sion
33
speci fi c crit eri a outl ined for each com ponent of Cl ass II cavi t y preparation as it appeared in the eval uati on f orm.
3.5 Dis cri min ati on Exercis e Rational e
Visual and t actil e exerci ses were desi gned in an effort to enhance identificati on of i deal perform ance of indivi dual procedural component and discrimi nation of variations from the ideal.
3.6 Dis cri min ati on Ex ercis e Design
Dis cri mination exercises were desi gned for 5 out of 6 com ponents that the pilot st ud y had identified as having poor i nt erexami ner reli abilit y The o5 com ponent s were 1) Proximal and Gi ngival Contact Cl earance, 2) Ret enti on Groove Placem ent, 3) Ret ention Groove Depth, 4) Preparation Walls and 5) Preparation M argin Ori ent ati on. A discussion of t he specific crit eri a for the component ‘Preparation Toilet’ was completed but no other discrimination exerci se was developed for this component . The des cripti on and desi gn of the five dis crim ination exercis es are as follows:
3.6.1. Proxi mal Con tact Clearan ce
34
mm on either one or both t he proximal wal ls as well as the speci fi cs of st udent perform ance t hat varied from the i deal and how t his was t o be ass ess ed.1 0
In an att em pt to fi nd an obj ective m eans of standardizing the ass es s ment of proxim al clearance, di git al im ages (Nikon D3100 cam era, Nikkor 105m m Lens, Si gm a EM -140 DG flash) of 82 P remier UNC 15 peri odont al probes were taken. The mean di am eter of the probes (mm ) was ass es sed usi ng Im age J ® (National Insti tut es of Heal th, B et hesda, Md) software.
The m ean di am eter of the UNC 15 probe was found to be 0.5 ( +0.02)m m at the 4-5 mm m ark and 0.75 ( +0.02)mm at the 11 -12 mm m ark (Fi gure 2.1). The UNC -15 peri odontal probe was therefore adopt ed as a st andardized m easurem ent device for the purpos e of ass essi ng proximal cont act cl earance. The discrimi nation exercise included a t actil e demonstrati on of how the proxim al contact cl earance is be ass es sed as ideal , if the diamet er of the periodont al probe at the 4/ 5mm mark (up to 0.5 m m) can not pas s bet ween t he proxim al surfaces , acceptabl e if the cl earance between the proximal surfaces was bet ween the 5 mm mark and the 11/ 12 mm m ark (between 0.5 mm and 0.75 mm ) and/or clini call y unacceptable, i f the cl earance between the proximal s urfaces is greater than the diamet er of t he periodontal probe at the 11/ 12mm mark (> 0.75 mm) bet ween t he proximal cont act cl earances (Fi gures 2.2 and 2.3).
3.6.2. Retenti on Groove Placemen t
35
shoul d t heoreti call y be placed approximat el y 0.2mm int ernal to the dentinoenam el junct ion (DEJ ) on t he faci al and lingual proximal preparation wal ls as this would ens ure sufficient amount of s upport ed enam el and avoi d injur y t o the pul p.1 0
Students m a y i ncorrectl y place ret ention grooves. Therefore t he discrimi nation exercise included a s eries of Cl ass II preparat ions wit h di fferent axial wall dept hs that represent ed the vari ous t ypes of errors found in student preparati ons . The dentiform teet h were s ectioned s agit all y so as to all ow visualization of t he gi ngi val and faci al external walls , the axial and pulpal internal wall s and various posi tions of ret enti on grooves (Fi gures 3.1 - 3.7). The preparati ons were as foll ows:
Preparation#1: Ideal axial wall dept h (0.5mm ins ide the DEJ ) with a retention groove in the enam el faci al wall i ndi cating a clinicall y unacceptabl e position (Fi gure 3.1).
Preparation#2: Ideal axial wall dept h (0.5mm ins ide the DEJ ) with a retention groove placed at the DEJ indi cati ng a cl inicall y unaccept abl e positi on (Fi gure 3.2).
Preparation#3: Ideal axial wall dept h (0.5mm inside the DEJ ) with ret ention gro ove pl aced ~0.2mm internal t o the DEJ indi cating a cl inicall y accept abl e positi on (Fi gure 3.3).
36
Preparation#5: Ideal axial wall dept h (0.5mm inside the DEJ ) with ret ention groove pl aced i n the gi ngival floor indicating a clini call y unaccept able positi on (Fi gure 3.5).
Preparation#6: Deep axial wal l (~2mm inside the DEJ ) wit h ret ention groove placed 0.2mm i nt ernal t o the DEJ indi cating a clinicall y accept able posit ion (Fi gure 3.6).
Preparation#7: Deep axial wall (2m m i nside the DEJ ) wit h ret enti on groove placed at t he axial and faci al wall li ne angl e indi cating a cli nicall y unaccept abl e positi on (Fi gure 3.7).
3.6.3. Retenti on Groove Depth :
Students ma y creat e a ret ention groove that has i ncorrect depth. The groove s houl d i deall y be 0.1 -0.5 mm deep to ensure adequat e ret ention.1 0
A di scrimi nat ion ex ercis e provi ded examiners wit h an expl orer and three sagi tal l y s ectioned dentiform t eeth, wit h Clas s II preparati ons, t hat cont ained ret ention grooves t hat were deep (> 0.5 mm i n depth), ideal (0.1 – 0.5 mm in depth) and shallow (< 0.1 m m in depth, undetect abl e). Deep or shall ow ret enti on grooves were deemed unaccept able . No di git al im ages were us ed to enhance discrimi nation of various level s of performance of t his component.
3.6.4. Preparation Finish
37
smoot hing of t he external wall s to eli minat e an y uns upport ed enam el and rounding t he int ernal line angl es to avoid stress concent rati on.1 0
A dis crimi nat ion exercise provided t he examiners with four Class II cavit y preparations that cont ai ned t he vari ous combinations of student perform ance of preparation wall finish. Each combi nation was identi fi ed as clini call y accept abl e or unaccept abl e. The preparation finis h com binations were as fol lows:
1) Preparation finis h resul ting in smooth wal ls and gent le transitions - clini call y accept abl e (Fi gures 4.1 & 4.2).
2) Preparation finis h resul ting in sm oot h wal ls and abrup t t ransitions -clini call y unaccept able (Fi gures 4.3 & 4.4).
3) Preparation fi nish resul ting i n rough walls and abrupt transitions - clini call y unaccept able (Fi gures 4.5 & 4.6).
4) Preparation fi nis h result ing i n rough walls and gentl e transiti ons - clini call y unaccept able (Fi gures 4.7 & 4.8).
3.6.5. Preparation Margin s:
38
average enam el rod ori ent ati on wil l result in cavosurface m argin angles that are approxim at el y 90°.1 0
Proximal ext ernal walls that are ori ented at 90° to a t angent ali gned with the external curvat ure of t he tooth are considered to have enam el that is support ed b y enamel . A 90° proximal wall ori entat ion is considered clini call y accept able. Proximal external walls that are ori ent ed < 80° t o a t angent ali gned with the external curvat ure of the t oot h are considered t o have enam el t hat is not support ed b y denti n . A proximal wal l orient ation that is < 80° is considered clini call y unaccept able. P roximal ext ernal walls that are ori ent ed > 100° to a tangent ali gned wit h the external curvature of the toot h will result in an am al gam restoration margin that is fragil e. A proximal wall orient ation that is > 100° is considered cl inicall y unaccept abl e.
Dis cri mination exercis es consist ed of di git al im ages of cl ini call y accept able and unaccept abl e proxim al wal l ori ent ati ons. A t ransparent prot ractor was superimposed over the images to enable more obj ective identi fi cation of wal l ori ent ati ons that were < 80°, were approximat el y 90° or were >100°. (Fi gures 5.1, 5.2 and 5.3). Exampl es of clini call y acceptabl e and unaccept abl e cavosurface m argin ori ent ati ons, i ncludi ng ideal cavos urface margins (90°) were pres ent ed and dis cus sed during t his dis cri minat ion exercis e.
3.7 Revi ew of the Asses s men t Rubri c
39
assess ment rubri c form was com pleted. Examiners were provi ded opportunit y to ask quest ions throughout the calibrat ion sess ion.
3.8 T i min g of th e Phase I and Phase II Assess ments 3.8.1 Ph as e I As sess men t
Each examiner was as ked t o assess t he 32 Class II cavit y preparations immediat el y aft er com plet ion of t he cal ibrat ion s es sion. The ass ess ment requi red approximat el y 2 -3 hours and was accom plished in one si tting.
3.8.2 Ph as e I I Ass es s ment
The exami ners as ses sed t he same 32 Cl as s II preparations aft er an average time i nt erval of 6 months. The ass essment was conduct ed usi ng t he s ame controlled settings as the Pilot and Phas e I studies. However, no cal ibrat ion sess ion was provided. The purpose of t he Phas e II ass essm ent was to eval uate the impact of t he pas sage of tim e on l evel s of examiner calibration.
3.9 O rgani zation of the Data
The Cl ass II preparation assessm ent poi nt values were transferred b y the prim ar y investi gato r from the ass essm ent rubri cs t o a di git al fil e (Tabl e 3). An independent investi gat or coded t he names of the exami ners using alphabet s letters from ‘A’ to ‘H’ so that the primary investigator was blinded to examiner identit y. The dat a was then subj ect ed t o st atisti cal anal ys is .
4. S tatistical Anal ys is
40
Pilot st ud y, Phas e I and Phas e II. The confidence i nt erval (C I) was cal culat ed for each component of t he cavit y preparation for all 3 ass es s ment s essi ons . 5. Results
5.1 Interexaminer Reliabil ity - Phase I
The int erex aminer reliabi lit y, reported as average percent age agreement, among t he examiners increas ed for 7 out of t he 13 component s when compared to the result s of the pilot stud y. However, for 3 com ponents (Adj acent Toot h Dam age, Unsupport ed Enam el and P rim ary/ Axi al Wall Dept h) the average percentage agreem ent did not change and for anot her 3 com ponents (Occlus al Convergence, R et ent ion Groove Pl acement and Ret ention Groove Dept h) there was a decline i n average percentage agreement (Fi gure 6).
Assessm ent of s om e com ponent s t argeted with dis crim inati on exerci ses showed increas e levels of facult y calibrati on whereas as sessm ent of other com ponent s di d not. The average percent age agreement i ncreas ed for P roxim al Cont act Cl earance from 52% before t he calibration s ess ion to 70% aft er
calibrat ion s essi on with 95% C I at [(68.79, 80.54)]. The average int erexami ner agreem ent for Retent ion Groove Pl acement decreas ed 58 t o 53% [95% C I
41
confidence i nt erval for each of the 13 component s aft er t he calibration s ess ion is reported in Tabl e 4.
5.2 Intraexamin er Rel iabili ty – Pil ot Study to Ph as e I
The average i nt raexaminer agreement am ong t he cours e facult y was 74 (+/ -5) %. The int raex aminer relia bi lit y (agreem ent of each exami ner wit h thems elves ) was calculated as average i ntraexaminer agreem ent (%) when com paring Pilot st ud y to Phas e I. The resul ts are li sted i n Table 5. Ass es sment of the 13 preparati on com ponent s vari ed greatl y from Pil ot t o Pha s e I. The component ‘Enamel Present’ had the lowest intraexaminer variation and the component ‘Proximal Contact Clearance’ had the highest intraexaminer variation (Figures 7 and 8). Table 6 shows the l evel of int ra -exami ner vari ation for each examiner for all 13 com ponent s bet ween the Pilot st ud y and Phase I.
5.3 Interexaminer Reliabil ity - Ph ase II
The int erex aminer agreem ent began to decline for 8 out of the 13 com ponent s aft er an average tim e interval of 6 m ont hs. For 3 components (Adjacent Tooth Damage , Ret ention Groove Pl acem ent and R et ention Groove Depth) t here was a sl ight i ncrease (1 -3%) in average percent age agreem ent among t he examiners . The average percentage agreement among the exami ners sta yed the sam e for one component (Occl usal Convergence). However, for 2 com ponent s (Is thmus Width and P reparati on Toil et) the i nt er examiner
42
‘Proximal Contact Clearance’ from 70% after calibration exercises to 60% [95% C I (53.38, 67.17)] after an average tim e i nterval of 6 -7 m ont hs. The average interexaminer agreement decreased from 58 t o 53% [95% C I (46.15, 59.97)] for ‘Retention Groove Placement’ and 57 to 56% [95% CI (51.03, 61.03)] for
‘Retention Groove Depth’. The average interexaminer agreement increased from 59 to 64% [95% CI (54.16, 68.62)] for ‘Preparation Wall Finish’, 54 to 63% [95% C I (58.13, 67.32)] for ‘Preparat ion Margin Ori ent ati on’ and 58 to 68% [95% CI (63.80, 72.81)] for ‘Preparation Debris’. The average percentage agreem ent and confi dence i nterval for as sessment of each component of the cavit y preparation during Phas e II is r eport ed in Tabl e 7.
5.4 Intraexamin er Rel iabili ty - Phas e I I
The average i nt raexaminer agreement am ong t he cours e facult y was 77 (+/ -7) % when compari ng P ilot stud y t o Phas e II and 76(+/ -8) % when
com paring Phas e I t o Phase II. The int raexaminer reli abilit y w as cal cul at ed for Pilot Stud y: Phas e I and Phas e I: Phas e II compari sons (Tabl es 8 & 9). Whil e some ex aminers rem ained consi stent i n t hei r ass es sment pat terns from Phas e I to Phas e II, t he ot hers reverted back to ass es sing wit h a variation of 30 -40% for a few components. Assessment of ‘Isthmus Width’ had the lowest intraexaminer variation and assessment of ‘Proximal Contact Clearance’ had the highest intraexaminer vari ati on (Fi gures 10 and 11). Comparis on of the i ntraexaminer vari ation that occurred whi l e assessi ng al l 13 components when comparing Phas e I: P hase II and the Pilot Stud y: P hase II and Phas e I: Phas e II
43
The l evel of int raexaminer vari ation (%) for all 13 components of Cl ass II cavit y preparation for each examiner when comparing Pilot st ud y t o Phas e II is shown i n Table 10 and the level of i ntraexaminer vari ation (%) for all 13
com ponent s of Cl as s II cavit y preparation for each exami ner when com pari ng Phas e I t o Phas e II is shown in Tabl e 11.
6. Dis cussi on:
It has been well docu mented that improving t he level of agreem ent am ong facult y m em bers is not an eas y task. 5 , 7 , 1 1. The over-arching goal of the current stud y was to determi ne the int erexaminer and int raexaminer reliabi lit y whi le assess ing 13 components of a preclini cal operative procedure com plet ed b y first year dent al students an d to s eek to increase facult y agreem ent in areas where it was low. The res ults of the pilot stud y confi rm ed t hat there were areas of l ow interexaminer agreement am ong the facult y. As part of the st ud y desi gn, s peci fi c exerci ses were devel oped and present e d i n a cali brati on session so as to increas e the abilit y of facult y to dis crim inate among various l evels of student
perform ance. The effici enc y of thes e exercis es was eval uat ed t hrough the use of immediat e (Phas e I) and del a yed (Phase II) int er - and i ntraexaminer reli abilit y testi ng.
44
one radiologist cat egoriz es a m amm ogram finding as normal and another
categorizes it as beni gn, but we do care i f one categorizes it as normal and the other as cancer1 2. Si milarl y for the current st ud y, i f we look at t he component Proximal contact clearance a disagreement between ‘No Clearance’ and ‘Open up to 0.5mm in all di re cti ons ’ i s not as severe as one bet ween ‘No Cl earance’ and ‘Open more than 0.75 mm in any direction’.
Proximal cont act cl earance i s t raditionall y identi fi ed b y the appearance of a visuall y open clearance at the proximal hei ght of contour. Asses sm ent of th e dist ance of proxim al cl earance is vague and s ubj ect to personal bi as.
Dimit rij evi c et al ex amined denti sts' and dental students ' abili ties to est imate small dept hs and di st ances and est ablis hed that i ndi vidual perceptual abiliti es var y widel y. This stud y hi ghli ght ed that s ome dentis ts and m an y dental students , parti cul arl y earl y i n thei r course, have great di fficult y in accurat el y gauging depths and dist ances.1 3 Cours e facult y must ensure st udent s receive objecti ve, unbi as ed as sessm ent of t he ir perform ance.
The res ults from Phase I showed t hat the interexaminer rel iabi lit y
improved with the us e of dis crimination exercis es. As part of t he dis crimination exerci ses , t he us e of UNC 15 peri odont al probe was int roduced in a novel wa y so as t o more obj ecti vel y ass ess proximal cont act clearance. The average percentage agreem ent am ong the exami ners i ncreased from 52% before
45
that int roduction of a s peci fi c instrum ent to ass es s thi s com ponent cont ribut ed to the i ncreas e in int erexami ner reli abili t y, and limit ed the i nflue nce of personal examiner bias associ at ed with the ass ess ment of this com ponent .
There was also an i ncreas e i n int erexami ner reliabilit y for component s such as ‘Isthmus Width’ and ‘Preparation Debris’ for which no discrimination exerci ses were desi gned. The authors specul at e t hat det ail ed discussions regarding the s peci fi c criteri a outli ned for all 13 com ponents of t he Cl ass II preparati on m a y have hel ped to limit mis -underst anding of s ubjective
interpretation of thes e criteri a.
The two com ponents for whi ch t here was no increase in reli abi lit y among the exam iners were ret ention groove placem ent and retention groove depth. Thi s was in spit e of exam iner parti cipati on i n carefull y desi gned di scriminati on exerci ses . Although studi es have been done on the si gni fi cance, ideal position and ideal depth of retenti on grooves, researchers have not been abl e to achieve consensus.1 1 - 1 3
A surve y b y David Moore in 1992 i nvesti gat ed the t eaching i n dent al schools regardi ng proximal retention grooves in Cl ass II cavit y prepa rations for am al gam restoration. He incl uded 64 schools across t he US and Canada. The response rat e was 92% (59 s chools) out of whi ch 61% (36 s chools ) of t he
schools responded ‘YES’ to teaching retention grooves and 39% (23 schools) of the schools respond ed ‘NO’ to t eaching t his t echni que1 4.
46
groove depth’. These increased variations prompted us to conduct an
informal2013 survey of the nation’s dental schools to inquire if they regularly taught t he pl acement of ret enti on grooves in cons ervati ve Cl as s II cavit y preparati ons for am algam rest orati on. The questionnaire was em ail ed t o 65 dental s chools across the nation. The respons e rat e was 53 (82%). Of the 53 schools that responded 24 of them replied ‘YES’ and 29 of them replied ‘NO’ to the teachi ng of ret ention groove s . It m a y be that vari ations i n the profes sional philosoph y (on the use of ret enti on grooves ) of t he ex aminers incl uded i n t his stud y limit ed the abi lit y to increase the l evel of agreem ent i n ass ess ing thes e com ponent s. It is im port ant to note that both t h e pl acem ent and accurate assess ment of ret enti on grooves i s di ffi cult.
Although there was a ris e in average percent age agreement am ong t he examiners aft er the calibration, t he result s of the P hase II grading session, reveal ed a definit e decline i n int erexa miner reli abi lit y. The st ud y did not test for the poi nt i n tim e when t he l evel of rel iabil it y start ed to decline but onl y det ected t hat there was a decli ne at an average tim e int erval of 6 months. This piece of i nform ati on is valuabl e and can be t ransl at ed as a need for frequent calibrat ion s essi ons t hroughout the academic year.
A limit ati on of thi s s tud y is t hat it does not evaluat e intraexaminer vari ation bas ed on the clini cal and t eachi ng experi ence of the examiners. J enkins et al found t hat the l evel o f pass –fail differences (i nt raexami ner vari abilit y) in their s tud y seem ed to be unrel at ed to the experi ence of t he
47
stud y rep orted that it is t heoreti call y poss ible that exami ner variabilit y alone ma y result i n some unfortunate undergraduat e s tudent fail ing on more t han one occasion parti cul arl y if acknowledged t hat on a di fferent da y, the same cavit y, assess ed b y the sam e exami ner, m a y well have pass ed8. An excerpt from
Mackenzie’s article on defining clinical competence in terms of quality,
quantit y and need for perform ance crit eri a, cl earl y st at es the problem of i ntra - examiner vari abilit y; “A dent al st udent report ed the following event. A good student and a poor s t udent w er e sitting next t o each other in a basi c t echni cs
labor atory. The good student finished the prepar ation in an ivorine t ooth and
took it to t he ins tructor. T he ins tructor s aid, “ fine work” , and gave him an ‘A’.
A little while later the poor student tooth the ‘A graded’ tooth to the same
instructor. The instructor looked at it, said, “Hm -mm, OK,” and gave him a
‘C’1 5 .
Anot her limit ation of this s tud y is that t he result s m a y be negativel y
infl uenced from examiner fati gue. Exami ner fati gue m a y pl a y a maj or role in the decline of int erexam iner reliabilit y. Havi ng to grade l arge number of sampl es at one tim e m a y cause t he exami ner to l ose focus. As suggest ed by Dhuru, R ypel and J ohnst on3 future studi es shoul d limit t he num ber of preparation s amples or have the examiners t ake frequent breaks aft er gradi ng 10 -15 s ampl es .
The res ults of the current st ud y indi cat e t hat dis criminati on exercis es are benefi cial for facul t y calibration. Future effort s in the fi eld of facult y
48
calibrat ion s essi ons. There i s a potenti al benefit i n des i gni ng calibrat ion
sess ions as online C E cours es, becaus e, t he y can be us ed b y s tudents and facult y as oft en as needed for s el f -anal ysis.
In summ ar y, the overall i nt erexami ner reli abili t y im proved aft er
calibrat ion. The res ults showed that, the obj ecti ve us e of an i nstrum ent (UNC 15 periodont al probe) increas ed int erexami ner rel iabilit y. The st ud y s upport s the use of di scrimi nation exercis es for facult y calibration in order to improve the consist enc y of facul t y -st udent communication. The st ud y findi ngs reveal a decreas e i n inter and int ra -examiner rel iabilit y at 6 m ont hs. Wide int raexami ner vari ation was not ed from P ilot St ud y to P hase I. Even though vari ations were not ed i n intraexaminer reli abi lit y aft er a t ime i nt erval of 6 months, the degree of vari ation was lower t han Pilot Stud y : P hase I.
7. Con clusion s:
- Overall int erexam iner rel iabilit y improved aft er cal ibrati on.
- The objecti ve us e of an inst rum ent (UNC 15 periodont al probe) increas ed interexaminer rel iabi lit y.
49
TABLE 1 - Class II Amalgam Procedure Performance Assessment Rubric Used For The Pilot Study, Phase I And Phase II
EXTERNAL OUTLINE
Caries Removal Complete removal at the DEJ Incomplete removal at DEJ
Isthmus Width Less than 1 mm
Between 1 mm and 1/3 of intercuspal distance Between 1/3 and 1/2 of intercuspal distance Greater than 1/2 of intercuspal distance Proximal Contact Clearance No Clearance
Open up to 0.5mm in all directions
Open between 0.5 and 0.75 mm in any direction Open more than 0.75 mm in any direction Adjacent Tooth Damage No damage
Requires re-contouring Requires restoration INTERNAL FORM
Enamel Present None
Less than or equal to 50% of preparation Greater than 50% of preparation
Primary Pulpal/Axial Wall Less than or equal to 0.5 mm internal to DEJ 0.5 - 1.5 mm internal to DEJ
2.0 - 2.5 mm internal to DEJ
Greater than 2.5 mm internal to DEJ Caries Removal Incomplete
Complete
Complete with excessive dentin removal RETENTION FORM
External Walls Occlusal convergence with ~90° cavosurface margins Excessive occlusal convergence with <90° cavosurface margins
External walls parallel
External walls diverge occlusally Retention Groove Placement Undermined enamel
≈ 0.2 mm internal to DEJ
Between 0.2 mm and 1 mm internal to DEJ Greater than 1mm internal to DEJ
Not Visible Retention Groove Depth Undetectable
Between 0.1 mm and 0.5 mm Greater than 0.5 mm
FINISHING
50
Supported enamel (90°) Enamel margin > 100° Preparation Toilet Debris present
Clean
51
TABLE 2 – Average percent agreement (mean +/- sd) and 95% CI for ass ess ment of co mpon ents of Clas s II cavi ty p reparation durin g the Pil ot study
COMPONENTS OF CLASS II CAVITY
PREPARATION MEAN
STD DEV
95% CI (LOWER, UPPER)
DEJ Caries Isthmus Width
Proximal Contact Clearance Adjacent Tooth Damage Enamel Present
Primary Pulpal Axial Wall Caries Removal
External Walls
52
53
54
55
56
57
FIGURE 3.3- 3D model demonstrating clinically acceptable retention groove placement ~0.2 mm internal to the DEJ such that it is partially in the dentin facial wall and
58
59
60
61
FIGURES 3.7- 3D model demonstrating clinically unacceptable retention groove
62
FIGURES 4.1 & 4.2 – 3D model demonstrating clinically acceptable preparation finish of smooth walls and gentle transitions. Figure 4.1
63
FIGURES 4.3 & 4.4 – 3D model demonstrating clinically unacceptable preparation finish of smooth walls and abrupt transitions.
Figure 4.3
64
FIGURES 4.5 & 4.6 – 3D model demonstrating clinically unacceptable preparation finish of rough walls and gentle transitions.
Figure 4.5
65
FIGURES 4.7 & 4.8 – 3D model demonstrating clinically unacceptable preparation finish of rough walls and abrupt transitions.
66
67
FIGURE 5.2- Image of the 3D model with a superimposed protractor that demonstrates clinically unacceptable preparation wall orientation of >1000.
68
FIGURE 5.3- Image of the 3D model with a superimposed protractor that demonstrates clinically acceptable preparation wall orientation of approximately 900.
69
TABLE 3 - Class II amalgam procedure assessment rubric that was used to define the level of student performance and associated point values for each component
PREPARATION COMPONENTS
SPECIFIC CRITERIA FOR EACH COMPONENT
POINT VALUES FOR EACH CRITERIA (Clinically acceptable-CA) (Clinically unacceptable-CU) EXTERNAL
OUTLINE
DEJ Caries Complete removal at the DEJ Incomplete removal at the DEJ
1 (CA)
0 (CU)
Isthmus Width
Less than 1 mm
Between 1 mm and 1/3 of intercuspal distance Between 1/3 and 1/2 of intercuspal distance
Greater than 1/2 of intercuspal distance 1 (CU) 3 CA) 2 (CA) 0 (CU) Proximal Contact Clearance No Clearance
Open up to 0.5mm in all directions
Open between 0.5 and 0.75 mm in any direction
Open more than 0.75 mm in any direction 1 (CU) 3 (CA) 2 (CA) 0 (CU) Adjacent Tooth Damage No damage Requires re-contouring Requires restoration 2 (CA) 1 (CU) 0 (CU) INTERNAL FORM Enamel Present None
Less than or equal to 50% of preparation
Greater than 50% of preparation 2 (CA) 1 (CU) 0 (CU) Primary Pulpal/Axial Wall
Less than or equal to 0.5 mm internal to DEJ
0.5 - 1.5 mm internal to DEJ
2.0 - 2.5 mm internal to DEJ Greater than 2.5 mm internal
3 (CA)
2 (CA)
70 to DEJ
Caries Removal
Incomplete
Complete
Complete with excessive dentin removal 0 (CU) 2 (CA) 1 (CU) RETENTION FORM External Walls
Occlusal convergence with ~90° cavosurface margin Excessive occlusal convergence with <90° cavosurface margin
External walls parallel External walls diverge occlusally 3 (CA) 2 (CA) 1 (CU) 0 (CU) Retention Groove Placement Undermined enamel
≈ 0.2 mm internal to DEJ Between 0.2 mm and 1 mm internal to DEJ
Greater than 1mm internal to DEJ Not Visible 0 (CU) 3 (CA) 2 (CA) 1 (CU) -1 (CU) Retention Groove Depth Undetectable
Between 0.1 mm and 0.5 mm
Greater than 0.5 mm
0 (CU)
2 (CA)
1 (CU) PREPARATION
FINISH
Preparation Walls Smooth, gentle transitions Rough, abrupt transitions
1 (CA)
0 (CU)
Preparation Margins
Unsupported enamel (< 80°)
Supported enamel (90°)
Enamel margin (> 100°)
0 (CU)
2 (CA)
1 (CU)
Preparation Toilet Debris present
Clean
0 (CU)
71
72
TABLE 4 – Average percent agreement (mean +/- sd) and 95% CI for assessment of components of Class II cavity preparation during Phase I
COMPONENTS OF CLASS II CAVITY
PREPARATION MEAN STD DEV
95% CI – PHASE I (LOWER, UPPER) DEJ Caries
Isthmus Width
Proximal Contact Clearance Adjacent Tooth Damage Enamel Present
Primary Pulpal/Axial Wall Caries Removal
External Walls
73
TABLE 5 - Average percentage intra examiner agreement for each examiner when comparing the results from Pilot Study to Phase I
EXAMINERS: PILOT STUDY – PHASE I
AVERAGE INTRA- EXAMINER AGREEMENT (%)
AVERAGE KAPPA
Examiner A Pilot - Phase I 68 0.32
Examiner B Pilot -Phase I 68 0.33
Examiner C Pilot -Phase I 74 0.50
Examiner D Pilot -Phase I 77 0.45
Examiner E Pilot -Phase I 72 0.49
Examiner F Pilot -Phase I 83 0.58
Examiner G Pilot -Phase I 72 0.41
74
75
