Dental education development reflection from an objective structured clinical examination

ORIGINAL ARTICLE

Dental education development reflection

from an objective structured clinical

examination

Stanley Shih-Li Tsai

, Jenny I-Chun Sar

,

Jenny Zwei-Chieng Chang

, Szu-Hsien Yu

,

Eddie Hsiang-Hua Lai

*

, Chun-Pin Lin

a

Department of Medical Education, National Taiwan University Hospital, Taipei, Taiwan

b

Graduate Institute of Medical Education and Bioethics, College of Medicine, National Taiwan University, Taipei, Taiwan

c

Department of Dentistry, National Taiwan University Hospital, Hsin-Chu Branch, Hsin Chu City, Taiwan

d

Department of Dentistry, National Taiwan University Hospital, Taipei, Taiwan

e_{School of Dentistry, National Taiwan University, Taipei, Taiwan}

Received 12 November 2014; Final revision received 22 December 2014

Available online 19 May 2015

KEYWORDS checklist; dentistry; examiner training; objective structure clinical examination

Abstract Background/purpose: Recently, dental education has put emphasis on the assess-ment of clinical competencies using an objective structured clinical examination (OSCE). The purpose of this study was to assess several clinical competencies required for dental grad-uates by having dental students and residents of different levels take the same set of OSCEs. Materials and methods: Twelve 5th-grade dental students (D5), 12 dental interns (Ri), and 12 1st_{-year dental residents (R1) were recruited to take the same set of OSCEs composed of six}

stations: Station 1, explanation of a treatment plan for restoration of a missing tooth; Station 2, taking a study cast impression with alginate; Station 3, explanation of a treatment plan for restoration of an inlay; Station 4, explanation to a mother for taking a radiograph of her child’s tooth; Station 5, placement of a rubber dam on a designated tooth and Station 6, explanation of pulpitis diagnosis and treatment. The examinee’s performance was graded using both global rating and checklist scores.

Results: There were significant differences in the mean global rating and checklist scores of Stations 3, 5, and 6 among the three groups. In Stations 3 and 5, Ri performed best, followed by R1 and D5. In Station 6, R1 performed best, followed by Ri and D5. In Stations 1, 2, and 4, there were no significant differences among the three groups. However, Ri performed better than R1 in Station 1.

* Corresponding author. School of Dentistry, National Taiwan University, Number 1 Changde Street, Jhongjheng District, Taipei 100, Taiwan.

E-mail address:[email protected](E.H.-H. Lai).

http://dx.doi.org/10.1016/j.jds.2014.12.001

1991-7902/Copyrightª 2015, Association for Dental Sciences of the Republic of China. Published by Elsevier Taiwan LLC. All rights reserved.

Available online atwww.sciencedirect.com

ScienceDirect

Conclusion: More emphasis should be placed on professional communication skills in the resi-dent training program.

Copyrightª 2015, Association for Dental Sciences of the Republic of China. Published by Else-vier Taiwan LLC. All rights reserved.

Introduction

The focus of dental education worldwide has expanded during the past few years. Discussions have ranged from core competencies of dentists and academic programs to

clinical assessment strategies.1e4 Current emphasis has

focused on patient-centered medical service and advanced clinical training, including especially the appropriate atti-tude and good communication skills. As a result, clinical performance assessment using the objective structured clinical examination (OSCE) has gained popularity. More-over, its usage in dental assessment has been broadly

dis-cussed.5e8 _{After analyzing the results of OSCE tests,}

academic courses could be adjusted to achieve the most suitable combination of teaching, testing, learning, and

performing.9,10

The purpose of this study was to assess several clinical competencies required for dental graduates by having dental students and residents of different levels take the same set of OSCEs. In the beginning, the teaching com-mittee raised several clinical competencies required for a dental graduate. However, no general consensus has been reached regarding the extent and degree of preparedness of these individualsdespecially considering that these are

students of different levels.11,12_{By utilizing OSCE,}

partici-pants of different levels were examined for certain clinical skills, and their performance outcomes were correlated with current dental education. Furthermore, by evaluating the variation between learning and performance outcomes, curriculum guidance can be re-established.

Materials and methods

An OSCE composed of six stations was designed for this study. Clinical scenarios were set up in the dental clinics of the National Taiwan University Hospital, Taipei, Taiwan where the examination took place. The process of station development began by assembling a multidisciplinary committee consisting of nine clinically well-experienced instructors from the Department of Dentistry, National Taiwan University Hospital. The committee provided a blueprint of certain core competencies required for a

dental graduate (Table 1). The development of the

ques-tionnaire was based on this blueprint. After generating the six OSCE stations, both reliability and validity were tested. Six clinical scenarios were set up, including four using standardized patients and two testing the clinical skills. In the four stations using standardized patients, participants were asked to give treatment plans for restoration of a missing tooth and an inlay, an explanation to a mother for taking a radiograph of her child’s tooth, and an explanation

of pulpitis diagnosis and treatment. In the two stations testing clinical skills, participants were asked to take a study cast impression with alginate and to place a rubber dam on a designated tooth. The examinees’ performance was graded using a checklist score (successful items/total items evaluated, expressed as a percentage) and a global

rating score (1Z clear fail; 2 Z borderline; 3 Z clear pass;

4Z very good pass; 5 Z excellent pass).13

Nine clinically well-experienced dental staff members were recruited as OSCE examiners. OSCE examiner work-shops were held once a week for 4 weeks. During the workshop, OSCE examiners were given a basic introduction about performance-based assessment and a more detailed instruction about the rating tool and the principle of formulating the checklists. Rehearsals were provided by video presentation and video rehearsal. Through team discussion, feedback, and educational programs, the con-fidence of the raters was enhanced. Furthermore, the

val-idity and consistency were improved.14In the examination,

residents of the dental department acted as standardized patients. A panel discussion was held 2 hours prior to the examination to improve the reliability of the actor’s performance.

Participants

Thirty-six participantsdconsisting of 12 5th-grade dental

students (D5), 12 dental interns (Ri, also called 6th_-year

dental students), and 12 1st-year dental residents (R1)d

were randomly selected and included in this study. Details on age, sex, and the past year’s average school grade are

listed inTable 2.

Methods

Thirty-six examinees were randomly assigned into six groups, with six participants in each group. Six 10-minute OSCE stations were designed: Station 1, explanation of a treatment plan for restoration of a missing tooth; Station 2, taking a study cast impression with alginate; Station 3, explanation of a treatment plan for restoration of an inlay; Station 4, explanation to a mother for taking a radiograph of her child’s tooth; Station 5, placement of a rubber dam on a designated tooth; and Station 6, explanation of pulpitis diagnosis and treatment. Each station provided 2 minutes for reading of the question and making the clinical judg-ment and 8 minutes for clinical performance. The whole procedure was recorded with a videocassette recorder. A faculty member was assigned in each station to monitor the whole procedure. Clinical performance was assessed using a checklist score and a global rating score, which were

previously formulated in the pretest meeting as a perfor-mance assessment in each clinical scenario.

Statistical analysis

The software SPSS version 11.0 for Windows (SPSS Inc., Chicago, IL, USA) was used for statistical analysis. The mean global rating and checklist scores among the three groups

of 12 5th-year dental students (D5), 12 dental interns (Ri),

and 12 1st_{-year dental residents (R1) were compared using}

multivariate analysis of variance, and the variances among the three groups were compared using Wilk’s lambda.

Results

As shown inTable 2, there was no significant difference in

baseline sex, age, and school grade among the three groups (D5, Ri, and R1). However, there were significant differ-ences in the mean global rating and checklist scores in Stations 3, 5, and 6 among the three groups with values of

PZ 0.0041, P Z 0.0003, and P Z 0.0003, respectively. In

Stations 3 and 5, Ri performed best, followed by R1 and D5. In Station 6, R1 performed best, followed by Ri and D5. In Stations 1, 2, and 4, there were no significant differences among the three groups. However, Ri performed better

than R1 in Station 1 (Table 3).

Discussion

The aim of the study was to evaluate whether there was an effective bridge between teaching or examining and prac-ticing, and to assess whether the lecture goal and core competence fully fulfilled clinical needs. We analyzed three groups of dental students before and after graduation by using the same setting of clinical scenario to see whether participants from different phases of dental edu-cation had different performances. The OSCE stations were set up according to the basic clinical skills that a dental student should acquire prior to graduation.

Theoretically, the 1st_{-year dental resident (R1) should}

perform better than the dental intern (Ri). However, in Station 1 (explanation of a treatment plan for restoration of a missing tooth), Station 3 (explanation of a treatment plan for restoration of an inlay), and Station 5 (placement of a rubber dam on a designated tooth), Ri performed best among the three groups. Stations 1, 3, and 4 comprised examinations using standardized patients, with the aim of analyzing professionality including attitude and communi-cation skills toward doctorepatient relationship. In partic-ular, in Station 3, wherein the dentist’s communication attitude was analyzed, R1 performed significantly worse than Ri. According to a previous study, this may be due to paying less attention and having busier routine work, and thus R1 neglected the importance of communication

atti-tude and skill.15 As a result, this reminds us that in the

resident training program after graduation, clinical

communication skills and attitude should be reemphasized. In Station 5, a clinical skill was examined. However, R1 performed significantly worse than Ri. This could be because this clinical skill is no longer as necessary as other

T able 1 Blueprint of the dental objective structured clinical examination in the School of Dentistry , National T aiwan University . Clinical competency Specialty P rosthodontics P eriodontics Family dentistry Operative dentistry Oral maxillofacial surgery P ediatric dentistry Endodontics Oral pathology Orthodontics P atient e dentist discourse (querying patients and responding to questions) + Basic physical examination + Basic technique ++ Oral education and postoperative explanations ++ + : Clin ical compe tenc y in this spec ialty .

core competencies as time passes by, and dentists are no longer familiar with this skill. Consequently, we should reconsider the necessity of requiring this clinical skill as a core competence, and narrow down the gap between lecture-based teaching and clinical practice.

In Station 2 (taking a study cast impression with alginate) and Station 6 (explanation of pulpitis diagnosis and treat-ment), R1 performed best, followed by Ri and D5; especially in Station 6, in which performance showed a significant difference. This reveals that the bridging between teaching or learning and practicing underlying these two stations has a good coordination. Moreover, the performance goals are set appropriately. We suggest that our results may be helpful in setting up other clinical core competencies.

There were no significant differences in performance in Stations 1, 2, and 4 among the three groups. This means that certain courses may be delivered in the earlier phase of dental education, such as in the clerkship, and higher standards of required competence can be given during the internship.

Considering the examination procedures and the grading systems, the global rating scores and the checklist scores were very similar, demonstrating that this examiner training program could efficiently train examiners to reach a certain standardized grading ability. Because of the limited number of participants, our results were not suit-able for further evaluation using the borderline regression method. However, our results could be used in discrimi-nating between any two of the three groups, and in building up a database or a reference using the Angoff method by a specialist committee after further reorganizing the diffi-culties and sorting them into an appropriate order.

In this study, we allowed dental students and residents of different levels to take the same set of OSCEs. The ex-aminees were assessed in an effort to measure their learning and clinical performance. With constructive feedback from the examinees, we can further develop a guideline in setting up the criteria and teaching goals that would be suitable for the different phases of dental edu-cation. Furthermore, our results indicate that the dental

Table 3 Means and standard deviations of the global rating scores (1Z clear fail; 2 Z borderline; 3 Z clear pass; 4 Z very good pass; 5_{Z excellent pass) and checklist scores (successful items/total items evaluated, expressed as a percentage) of} dental objective structured clinical examination for 5th-year dental students (D5; nZ 12), dental interns (Ri; n Z 12), and 1st -year dental residents (R1; n_{Z 12).}

Stationb Mean global rating scoresa Mean checklist score (%)a

D5 Ri R1 D5 Ri R1 P

Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD

1 3.00_{ 1.21} 4.17_{ 0.83} 3.67_{ 1.07} 75.00_{ 10.98} 81.48_{ 9.28} 80.09_{ 10.45} 0.1262 2 3.42 0.79 3.92 0.51 4.08 0.51 56.67 16.14 64.17 13.79 70.00 13.48 0.1165 3 1.92 0.67 3.50 1.00 2.67 0.98 45.08 21.97 67.43 18.06 59.09 17.87 0.0041* 4 3.33 0.78 3.75 0.75 3.92 0.79 68.75 13.67 82.50 7.83 79.17 15.93 0.0739 5 2.75 0.75 3.92 0.29 3.75 0.97 70.37 14.67 87.04 6.41 78.24 11.95 0.0003* 6 2.08_{ 1.00} 3.25_{ 0.87} 3.33_{ 0.65} 51.25_{ 15.97} 58.33_{ 15.28} 60.83_{ 10.41} 0.0003* SDZ standard deviation. * Significant difference.

a_{The mean global rating and checklist scores among the three groups of D5, Ri, and R1 were compared using multivariate analysis of}

variance, and the variances among the three groups were compared using Wilk’s lambda with P_{< 0.05.}

b _{Station 1Z explanation of a treatment plan for restoration of a missing tooth. Station 2 Z taking a study cast impression with}

alginate. Station 3_{Z explanation of a treatment plan for restoration of an inlay. Station 4 Z explanation to a mother for taking a} radiograph of her child’s tooth. Station 5_{Z placement of a rubber dam on a designated tooth. Station 6 Z explanation of pulpitis} diagnosis and treatment.

Table 2 Demographic data of 5th-year dental students (D5; nZ 12), dental interns (Ri; n Z 12), and 1st-year dental residents (R1; nZ 12). D5 Ri R1 Percentage (%) Age (y) 20e23 9 1 0 27.8 24e27 3 11 10 66.7 28e31 0 0 2 5.5 Sex Male 7 6 6 52.8 Female 5 6 6 47.2

Average school grade 83.6_{ 8.6} 86.2_{ 10.3} 87.4_{ 12.7}

resident training program should focus more on professional communication skills.

Conflicts of interest

All contributing authors declare no conflicts of interest.

Acknowledgments

This work was supported by grants (NSC 102-2314-B-002 -057 -MY3) from National Science Council, Taiwan and from Professor Hsueh-Wan Kwan Faculty Development Founda-tion from the NaFounda-tional Taiwan University Dental School Alumni Association, Taiwan. We thank the participating experts for their time and effort. We also thank the Na-tional Taiwan University Hospital, School of Dentistry, Department of Dentistry, and Department of Medical Edu-cation for providing valuable resources and a venue for this research.

References

1.Gerrow JD, Murphy HJ, Boyd MA. Review and revision of the

competencies for a beginning dental practitioner in Canada.

J Can Dent Assoc 2007;73:157e60.

2.Cowpe J, Plasschaert A, Harzer W, Vinkka-Puhakka H,

Walmsley AD. Profile and competences for the graduating Eu-ropean dentistdupdate 2009. Eur J Dent Educ 2010;14:

193e202.

3. Australian Dental Council. Professional attributes and compe-tencies of the newly qualified dentist. Australian Dental Council June 2010. Available at:http://www.ada.org.au/App_ CmsLib/Media/Lib/1003/M221764_v1_634054490286100138. pdf. [date accessed 02.05.14].

4.ADEA foundation knowledge and skills for the new general

dentist: (as approved by the 2011 ADEA House of Delegates).

J Dent Educ 2013;77:903e7.

5. Yip HK, Smales RJ. Review of competency-based education in

dentistry. Br Dent J 2000;189:324e6.

6. Zartman RR, McWhorter AG, Seale NS, Boone WJ. Using

OSCE-based evaluation: curricular impact over time. J Dent Educ

2002;66:1323e30.

7. Curtis DA, Lind SL, Brear S, Finzen FC. The correlation of

stu-dent performance in preclinical and clinical prosthodontic

as-sessments. J Dent Educ 2007;71:365e72.

8. Schoonheim-Klein M, Muijtens A, Habets L, et al. On the

reli-ability of a dental OSCE, using SEM: effect of different days.

Eur J Dent Educ 2008;12:131e7.

9. Gamboa-Salcedo T, Martinez-Viniegra N, Pen˜a-Alonso YR,

Pacheco-Rı´os A, Garcı´a-Dura´n R, Sa´nchez-Medina J. Objective

structured clinical examination as an instrument for evaluation of clinical competence in pediatrics: a pilot study. Bol Med

Hosp Infant Mex 2011;68:169e76.

10. Hamann C, Volkan K, Fishman MB, Silvestri RC, Simon SR,

Fletcher SW. How well do second-year students learn phys-ical diagnosis? Observational study of an objective struc-tured clinical examination (OSCE). BMC Med Educ 2002;2:

1e11.

11. Hsu TC, Tsai SSL, Chang JZC, Yu SH, Lai EHH, Lin CP. Core

clinical competencies for dental graduates in Taiwan: considering local and cultural issues. J Dent Sci 2014;10(2):

161e6.

12. Laan RFJM, Leunissen RRM, van Herwaarden CLA. The 2009 framework for undergraduate medical education in the Netherlands. GMS Z Med Ausbild 2010;27:Doc35. http:

//dx.doi.org/10.3205/zma000672.

13. Cunnington JPW, Neville AJ, Norman GR. The risks of

thor-oughness: reliability and validity of global ratings and

check-lists in an OSCE. Adv Health Sci Educ 1997;1:227e33.

14. Lin CJ, Chang JZC, Hsu TC, et al. Correlation of rater training

and reliability in performance assessment: experience in a

school of dentistry. J Dent Sci 2013;8:256e60.

15. Feudtner C, Christakis DA, Christakis NA. Do clinical clerks

suffer ethical erosion? Students’ perceptions of their ethical environment and personal development. Acad Med 1994;69: