Using an Audience Response System to Prepare Athletic Training Students for the Board of Certification Exam

(1)

Athletic Training Education Journal ⎢ Volume 7 ⎢ Issue 4 ⎢ October-December 2012 198 © National Athletic Trainers’ Association

www.nataej.org ISSN: 1947-380X

E

DUCATIONAL

T

ECHNIQUE

Using an Audience Response System to Prepare Athletic

Training Students for the Board of Certification Exam

Kelly Potteiger, PhD*†, Andrew Lundgren, MEd†

*Rocky Mountain University of Health Professions, Provo, UT, †North Park University, Chicago, IL

Dr. Potteiger is currently an assistant professor of athletic training/clinical coordinator at North Park University. Please address all correspondence to Kelly Potteiger, PhD, North Park University, 3225 West Foster Avenue, Box 25, Chicago, IL 60625. [email protected].

Full Citation:

Potteiger K, Lundgren A. Using an Audience Response System to Prepare Athletic Training Students for the Board of Certification Exam. Athl Train Educ J. 2012;7(4):198-204.

Objective: To describe a pedagogical method using an Audience Response System (ARS), commonly

re-ferred to as “clickers,” for use in preparing athletic training students (ATSs) for the Board of Certification (BOC) exam.

Background: The BOC is the only certifying agency for athletic trainers in the United States. ATSs may

be-come a certified athletic trainer (AT) upon graduating from a Commission on Accreditation of Athletic Training Education (CAATE) accredited Athletic Training Education Program (ATEP) and successfully passing the BOC exam. In 2012, CAATE standards were revised to require programs to publish their graduates’ recent exam scores on their website (in aggregate form).

Description: Preparation for the BOC exam can be a monumental and stress-inducing task for the ATS due to

the plethora of information to review. Most ATEPs include test preparation assistance within the curriculum. To date, a review of the literature revealed no published documentation of successful test preparation strategies.

Clinical Advantages: The data collected from use of the ARS during BOC exam preparation can be a valuable

tool in identifying domain areas in which an ATS may be deficient. In doing so, an ATS may redirect study time to focus on weaker concepts. Additionally this data, when trended, can provide invaluable programmatic infor-mation when used as part of a larger assessment plan. As part of the revised CAATE standards, programs are also required to meet or exceed a score of 70% for the first-time pass rate. As a result, there may be a heightened need for test preparation strategies.

Conclusion: Athletic training educators seeking a pedagogical method for BOC exam preparation may benefit

from the implementation of an ARS to increase ATS motivation and accountability. As an additional benefit, programmatic educational assessment data may also be collected.

Key Words: Clickers, Test Preparation, Pedagogy

(2)

Table

of

Con-tents

Introduction

Preparation for the Board of Certification (BOC) exam can be a monumental and a stress-inducing task for the athletic training student (ATS) due to the plethora of information to review. An athletic trainer can only become nationally certified upon graduating from a Commission on Accreditation of Athletic Training Ed-ucation (CAATE) accredited Athletic Training Educa-tion Program (ATEP) and successfully passing the BOC exam. The purpose of this exam is to protect the public by verifying the certified athletic trainer (AT) is qualified to practice clinically.1 While the need for such an exam is clear, a ATS’s future professional plans may hinge on passing the examination. This pressure often adds to a ATS’s pre-test anxiety levels.

ATEP faculty and staff members, both clinical and di-dactic, have the responsibility to ensure ATSs are suf-ficiently prepared both for the BOC exam and suc-cessful employment as ATs. The first-time national pass rate for the 2010-2011 BOC exam was 60.7%.2 While these scores are trending upward, they still re-main below other health professions’ first-time certifi-cation rates.3,4_{In 2012, CAATE released a new} edi-tion of the Standards for the Accreditaedi-tion of Profes-sional Athletic Training Programs.5_{In this revised} edi-tion, 2 new standards were added in the effort to in-crease transparency in each program’s BOC exam performance. Standard IIB2a, requires programs to post their most recent 3 years of BOC exam aggre-gate data on a publicly accessible area of the gram’s website. While Standard IID2 requires pro-grams to meet or exceed a 3-year aggregate of 70% for their first-time pass rate. The addition of these two standards may heighten the need for test preparation strategies.

In an effort to increase exam preparation, some ATEPs require ATSs to complete their own qualifying exami-nation before the program director will sponsor the ATS for the BOC exam. However, these tests are of-ten produced by the institution and may not provide the ATS with a detailed analysis of his/her strengths

Using an Audience Response System to Prepare Athletic

Training Students for the Board of Certification Exam

Kelly Potteiger, PhD, Andrew Lundgren, MEd

and weaknesses in each educational domain. This practice also raises some questions such as: 1) Do ATSs perform better on the BOC exam with multiple opportunities to take a similar exam? and 2) How does providing the ATS with information on his/her strengths and weaknesses in each of the BOC Role Delineation Study/Practice Analysis (RDS/PA) Domains affect first-time pass rates? In our experience, the implementa-tion of 3 distinct directives positively influenced ATS performance. These directives are: 1) Repeatedly chal-lenging the ATS with questions similar to those on the BOC exam, 2) Providing an opportunity for discussion of each question immediately following the ATS’s re-sponse, and 3) Reinforcing this performance with quantitative data regarding his/her strengths and weak-nesses in each of the athletic training domains. There-fore, our purpose is to describe a pedagogical method using these directives to assist in preparing ATSs for the BOC exam. Although there are many ways to in-corporate these directives, in our ATEP, we used an audience response system (ARS) to assist in ATS en-gagement and to streamline the data collection pro-cess.

Audience Response System

Commonly referred to as “clickers,” the ARS is a com-puter-based system that allows for student interaction during a traditional course lecture using wireless key-pads. There are 3 main components to an ARS: 1) A receiver, 2) Wireless keypads the students use to in-teract with the instructor(the keypads utilize either ra-dio frequency or infrared technology to transmit their signals)and 3) Computer software needed to organize and run the system. Depending on the brand of ARS, the software may be embedded into Microsoft PowerPoint©_{. Some benefits of an ARS include the} ability to assign each student his/her own clicker, di-vide students into groups, and upload data directly into learning management systems (eg, Blackboard, WebCT) and spreadsheets (eg, Excel). While there is well-documented evidence for use of ARS in the class-room,6-9_{we did not find published documentation for}

(3)

use of an ARS in preparation for a professional certi-fying exam.

Exam Format

The BOC exam consists of a combination of multiple choice, interactive (drag and drop, text-based simula-tion) and scenario-based questions.1_{The exam} con-tent is based on the BOC RDS/PA which examines the current knowledge, skills, and abilities required for entry-level ATs.10_{The RDS/PA organizes this material} into specific content areas or “domains” (Table 1). Of note, the following pedagogical method was designed based on the 5th_{edition of the RDS/PA. Since that time,} a new 6th_{edition of the document was released. ATEP} faculty members who wish to use this method are urged to alter their approach to better reflect the latest edition.

Test Preparation

ATSs prepare for the BOC exam through the didactic and clinical components of the ATEP. In addition, there are numerous resources available to assist the ATS in their preparation. Some examples include preparato-ry workshops, exam books, DVD’s, and flash cards. The BOC offers self-assessment exams to candidates in the effort to assist in their preparation. These ex-ams are designed by content experts using the same standards as the BOC exam.1_{The BOC recommends} ATSs attempt at least one self-assessment exam as this experience is most similar to taking the actual exam. However, one study suggests there is no

dif-Table 1. BOC Role Delineation Study/Practice Analysis Domains 1*

Domain % on

Exam

Prevention 15.72

Clinical Evaluation and Diagnosis 22.91

Immediate Care 17.50

Treatment, Rehabilitation, and

Reconditioning 23.30

Organization and Administration 11.29

Professional Responsibility 9.27

* Domain names have changed since this data were collected.

Table 2. Course Outline

Week Weekly Topics/Assignments

1 Overview of BOC exam

Create study guide Register for BOC exam 2 Review study calendar

Clicker test on Domain II: Clinical Evaluation and Diagnosis

Due: self-assessment test results for written test

Due: study guide calendar

3 Clicker test on Domain II: Clinical Evaluation and Diagnosis

4 Clicker test on Domain IV: Treatment, Rehabilitation and Reconditioning 5 Clicker test on Domain IV: Treatment,

Rehabilitation and Reconditioning

6 Clicker test on Domain III: Immediate Care 7 Clicker test on Domain III: Immediate Care

Due: 2nd_{attempt of self-assessment test} (optional)

8 Clicker test on Domain V: Organization and Administration

9 Clicker test on Domain I: Prevention 10 Clicker test on Domain VI: Professional

Responsibility

Due: 3rd_{attempt of self-assessment test} (optional)

11 Clicker test on all domains 12 Clicker test on all domains

ference in the first-time pass rate of those ATSs who attempt multiple BOC self-assessment exams and those who do not attempt the self-assessment.11_In addition to ATS self-preparation methods, some ATEPs choose to imbed forms of test preparation within their curriculum. At our institution, a two-hour senior capstone course is dedicated to assist ATSs in honing their focus on their upcoming BOC examination. The Capstone Experience

The Student’s Role

The capstone course is 16 weeks in duration and tak-en during the spring of the ATSs’ stak-enior year. At the start of the course, ATSs were charged with creating an individualized study calendar using the course syl-labus as a template (Table 2). This calendar desig-nated which domain the ATSs needed to focus on each week. Each ATS was tasked with identifying the days/ times as well as the specific content he/she needed to

(4)

Table

of

Con-tents

Year 1 Self Assessment 1 (by domain) Domain II-Week 1 Domain II-Week 2 Weekly

I II III IV V VI % # Correct/ # Possible % # Correct/ # Possible % % Student Ab 0.25 0.64 0.38 0.42 0.67 0.80 0.50 0/32 0 22/48 0.46 NA Student B 0.63 0.82 0.88 0.83 0.67 0.80 0.78 24/32 0.75 40/48 0.83 8% Student C 0.50 0.82 0.75 0.75 0.67 1.00 0.74 21/32 0.66 37/48 0.77 12% Student D 0.88 0.73 0.38 0.58 0.50 0.60 0.62 17/32 0.53 35/48 0.73 20% review. The course instructor recommended ATSs

al-lot 10-12 hours per week for the 3 months immediate-ly preceding the exam for their preparation. ATSs were also encouraged to consult both the RDS/PA10_{and the} syllabi from all previous major coursework to ensure all the necessary content for each domain was em-bedded into their calendar. Students then organized the material into specific units (Table 3). The object of the study calendar was to provide the ATS with an overall plan of action. This plan provided both direc-tion and accountability to the ATS for what (i.e., con-tent) and when (i.e., day/time) to study. Adherence to the calendar was essential since the in-class activities corresponded to the domain listed in both the syllabus and study calendar.

Completion of at least one BOC self-assessment exam was required at the start of the course; although, ATSs may choose to attempt the remaining self-assessment exams available from the BOC multiple times through-out the semester (Table 2). Feedback from the BOC on the self-assessment exam occurred in one of two ways:12_{1) The ATS may choose to take the exam in}

“study mode” or 2) “test mode.” In “study mode” the self-assessment is not scored. Instead, the ATS re-ceived feedback on the exact questions missed as well as a list of references for the ATS to review. In “test mode” the exam was scored; therefore, the ATS did not receive information on the specific questions missed. Instead, the ATS was provided the percent-age correct from each domain. ATSs in the course were required to submit the self-assessment report from “test mode” during the second week of the course. ATS were then encouraged to adjust their study calendar based on the feedback provided by the exam(s).

The Instructor’s Role

One of the instructor’s primary roles was to design an ARS quiz for use each week during weeks 2-12 of the course. Each week’s 30-40-question quiz focused on the domain specified in the course syllabus. Each question was assigned to the appropriate domain for organizational purposes. This process is difficult as most questions could be classified into more than one domain. The instructor preparation was extensive dur-Table 3. Athletic Training Student Study Calendar- Week Two

Study Time Area Specific Focus

Monday (11-12) Foot and Ankle Anatomy, injuries, special tests

Wednesday (2-4) Knee Anatomy

Saturday (10-12) Knee Injuries and Special Tests

Hip Anatomy

Sunday (2-4) Hip Injuries and Special Tests

(Time of day)

Table 4. Year One Organization of Dataa

a_{All data is fictional for explanatory purposes only.} b _{Student passed on first attempt.}

(5)

Table 5. Year Two Organization of Dataa

Self Assessment 1 (by domain) Domain II-Week 1 Domain II-Week 2 Weekly

I II III IV V VI % # Correct/ # Possible % # Correct/ # Possible % % Year 1 Student A 0.25 0.64 0.38 0.42 0.67 0.80 0.50 0/32 0 22/48 0.46 NA Student Bb 0.63 0.82 0.88 0.83 0.67 0.80 0.78 24/32 0.75 40/48 0.83 8% Student C# 0.50 0.82 0.75 0.75 0.67 1.00 0.74 21/32 0.66 37/48 0.77 12% Student D 0.88 0.73 0.38 0.58 0.50 0.60 0.62 17/32 0.53 35/48 0.73 20% Year 2 Student E 0.63 0.73 0/63 0.42 0.33 0.40 0.54 16/32 0.50 27/48 0.56 6% Student F 0.88 0.64 0.75 0.67 0.50 1.00 0.72 22/32 0.69 40/48 0.83 15% Student G 0.63 0.73 0.50 0.75 0.50 0.60 0.64 19/32 0.59 33/48 0.69 7% Student H 0.38 0.55 0.63 0.50 0.67 0.80 0.56 24/32 0.75 27/48 0.56 -19% a

All data is fictional for explanatory purposes only.

b

Student passed on first attempt.

ing the first year the system is implemented. When developing their own ARS quizzes, course instructors may choose to write their own questions or use tions from published sources. Multiple choice ques-tions tend to work best in this format.

In-Class Activities

To implement this process, ATSs were assigned their own clicker. Each clicker possesses its own unique code, which is typically provided on the back of each keypad. The ATS and their designated code were en-tered into the ARS software. Once this was complet-ed, the software tracked each ATS’s response. This information was submitted to the instructor but re-mained anonymous to the members of the class. Af-ter each week’s ARS quiz, the results were automati-cally uploaded to the instructor’s computer (Table 4). ATSs were presented each week’s ARS quiz using presentation software ( Microsoft PowerPoint©_{). Only} one question was presented per slide. The ATS an-swered the question using their keypad. The ATS was allowed to change his/her response as long as polling remained open. The instructor may choose to close

polling in a variety of different ways: 1) Based on a pre-determined length of time (eg, 10 seconds), 2) once all ATSs respond or 3) once the ATSs reach a consensus. In our case, we chose not to close polling until after all the ATSs responded. This allowed each ATS ample time to think about his/her response. Once the polling closed, a bar graph appeared depicting the percentage of ATSs who picked each of the possible answers. The key was to discuss the rationale for the correct and/or incorrect answer(s) to each question. The ability to provide real-time feedback was a dis-tinct advantage from other types of traditional test prep-aration (eg, computer based exams). The feedback provided was tailored to the ATSs’ response(s) and allowed the instructor the opportunity to discuss in detail why certain responses were correct/incorrect while the ATS was still focused on the particular ques-tion. The amount of discussion time needed varied according to: 1) Percentage of ATSs who selected the correct answer and 2) distribution of the incorrect answers (ie, 75% of students chose 1 alternative an-swer rather than the correct anan-swer). In cases where the students unanimously answer a question correct-ly, there may be little reason to pause for feedback and the instructor can simply advance to the next

(6)

Table

of

Con-tents

as the timing of feedback influences ATS test prepara-tion.

Reflections

Our experience using this type of pedagogical meth-od revealed that ATSs exited the course with a deeper understanding of the content and improved test taking ability. ATS perceptions of the course design were also very positive. The first class to use this method report-ed they enjoyreport-ed the course format. However, they did not trust the process until after they received the re-sults of their BOC exam. Future cohorts engaged im-mediately due to the success of the previous years’ classes. In addition, the advantage of a comparison of their clicker scores compared to the previous cohort helped manage ATS anxiety. ATSs reported they ap-preciated the accountability both anonymously via the clickers and directly through the generated reports. When used on a yearly basis, this pedagogical tech-nique also served as a tool for programmatic assess-ment. Since each week focused on a selected domain of Athletic Training (Table 2), yearly trends started to appear that were used to identify areas of strength and weaknesses in the curriculum. Faculty members were encouraged to review the trended data in an ef-fort to identify weak areas in the curriculum and imple-ment changes accordingly.

Acknowledgements

The audience response system was purchased with an individual project development grant issued by North Park University, Chicago, IL. The authors deny any competing financial, professional or personal in-terests that might have influenced the performance or presentation of the work described in this manuscript. Thank you to Dr. Malissa Martin for her guidance with this publication.

References

1. BOC exam: candidate handbook. In: BOC, ed. Omaha, NE2010:36.

2. Johnson B. Examination review for 2009-10 testing year: board of certification (BOC) ex-amination for athletic trainers: Board of Certi fication;2010.

3. NCSBN. NCLEX statistics for NCSBN. 2010;

https://http://www.ncsbn.org/Table_

tion. Therefore, the amount of questions reviewed each class period varied and was based upon the amount of time ATSs required to answer the question as well as the discussion that ensued.

Quantitative Feedback

Student handouts, in the form of a table, were provid-ed each week (Table 4). The data providprovid-ed in this hand-out included the ATS’s self-assessment score (taken the second week of class) for the week’s domain of focus as well as the results of the previous week’s ARS quiz. This information was also saved for purposes of programmatic review and was used to access trends on a yearly basis.

During year 2 of this technique, we adjusted the ATS handout to include blinded data from the same week of the previous academic year. Notations marked the scores of the ATSs whom passed the BOC exam on their first attempt (Table 5). This data was particularly useful in helping ATSs understand how they compare to ATSs in previous cohorts. The scores of ATSs who passed the BOC exam on the first attempt were par-ticularly impactful. The data also helped to reduce the anxiety of ATSs who were scoring well; at the same time, it helped motivate the ATSs who scored poorly by refocusing their attention on weaker areas. Weak-er areas wWeak-ere defined by the benchmark of success set by previous cohorts. It also allowed the instructor to determine if the ATS was studying the correct breadth and depth of the material. If not, the instructor suggested additional resources for the ATS.

Limitations

There were some concerns among the ATEP faculty regarding the quality of the questions used. Since the questions were either developed in-house or via pub-lished study guides, they were not of the same quality as the BOC exam questions. However, since the same questions were used each year, it provided excellent institutional data. Of additional concern, the questions may not be assigned to the appropriate domain. This may have resulted in ATSs allotting additional time to some domains unnecessarily. Finally, this pedagogi-cal method has only been implemented in the ATEP in which it was developed; therefore, results may not be generalizable to other ATEPs. Future research should investigate how various modes of discussion as well

(7)

www.nataej.org

ISSN: 1947-380Xof_Pass_Rates_2010.pdf. Accessed 12/21/_2010. 4. FSBPT. NPTE Pass Rate. 2010; bpt.org/

ForCandidatesAndLicensees/NPTE/ PassRates/index.asp#ExamYear. Accessed 12/21/10.

5. Standards for Accreditation of Educational Programs for the Professional Preparation of the Athletic Trainer: CAATE;2012.

6. Brewer C. Near real-time assessment of stu-dent learning and understanding in biology courses. Bioscience. 2004;54(11):1034-1039. 7. Drabenstott K. Interactive multimedia for

li-brary-user education. Portal Libr. Acad. 2003;3(4):601-610.

8. Skiba D. Got large lecute hall classes? Use clickers. Nursing Education Perspective. 2006;27(5):278-280.

9. Duncan D. Clickers: a new teaching aid with exceptional promise. AER. 2006;5(1):70-88. 10. BOC. Role Delineation Study. Omaha, NE:

Board of Certification;2004.

11. Butterfield D. Do exam candidates in athletic training who take one or more board of certifi-cation self-assessment examinations have a significantly higher pass rate on the exam than those who do not?: New Mexico State Univer sity 2010:136.

12. BOC. Self assessment exams. http:// www.bocatc.org/index.php?option= com_ content&view=article&id=136&Itemid=149.