MEDICAL
EDUCA
To ensure that the Comprehensive Osteopathic Medical Licensing Examination- USA (COMLEX-USA) reflects the evolving practice of osteopathic medi-cine, the National Board of Osteopathic Medical Examiners has developed new content and format specifications for an enhanced, competency-based examination program to be implemented with COMLEX-USA Level 3 in 2018. This article summarizes the evidence-based design processes that served as the foundation for blueprint development and the evidence supporting its validity. An overview is provided of the blueprint’s 2 dimensions: Competency Domains and Clinical Presentations. The authors focus on the evidence that supports interpretation of test scores for the primary and intended purpose of COMLEX-USA, which is osteopathic physician licensure. Important sec-ondary uses and the educational and catalytic effect of assessments are also described. This article concludes with the National Board of Osteopathic Medical Examiners’ plans to ensure that the COMLEX-USA series remains current and meets the needs of its stakeholders—the patients who seek care from osteopathic physicians.
J Am Osteopath Assoc. 2017;117(4):253-261 doi:10.7556/jaoa.2017.043
Keywords: COMLEX-USA, competencies, licensure
From the National Board of Osteopathic Medical Examiners in Conshohocken, Pennsylvania (Drs Gimpel, Horber, and Sandella), the University of North Texas Health Science Center in Fort Worth (Dr Knebl), and the Michigan State University College of Osteopathic Medicine in East Lansing (Dr Thornburg). Financial Disclosures:
None reported. Support: None reported. Address correspondence to Dorothy Horber, PhD, National Board of Osteopathic Medical Examiners, 101 W Elm St, Ste 100, Conshohocken, PA 19428-2003. E-mail: dhorber@nbome.org Submitted September 21, 2016; revision received December 13, 2016; accepted January 5, 2017.
Evidence-Based Redesign
of the COMLEX-USA Series
John R. Gimpel, DO, MEd; Dorothy Horber, PhD; Jeanne M. Sandella, DO; Janice A. Knebl, DO; and John E. Thornburg, DO, PhD
JAOA/AACOM
A
s a licensing examination, a primary purpose of the Comprehensive Osteo-pathic Medical Licensing Examination-USA (COMLEX-USA) is to assure the public that candidates who have passed the examination series have met an established national standard of osteopathic medical knowledge and clinical skills, making them eligible to apply for the privilege of practicing osteopathic medicine in all 50 states. Examination scores have also been used for important secondary purposes, such as a screening tool for residency program directors1 and as a graduation requirement. Since their initiation in 1995, the COMLEX-USA series has been developed around a patient presentation–based, 2-dimensional blueprint2 that has since been used by other medical licensing organizations.3,4 The Federation of State Medical Boards endorsedMEDICAL
EDUCA
into supervised practice (ie, graduate medical education), and Decision Point Two was defined as a series of post-doctoral assessments leading to unsupervised practice (ie, eligibility for licensure). The Panel included repre-sentatives from the American Osteopathic Association (AOA), American Association of Osteopathic Exam-iners, Federation of State Medical Boards, American Association of Colleges of Osteopathic Medicine (AACOM), American Colleges of Graduate Medical Education (ACGME), Organization of Program Director Associations, Association of Osteopathic Directors and Medical Educators, Educational Council on Osteopathic Principles, Educational Commission for Foreign Med-ical Graduates, NBOME Board of Directors, and NBOME National Faculty.
Further, the Coalition for Physician Accountability, of which the NBOME is a charter member, along with the AOA and AACOM, issued the Consensus Statement on a Framework for Professional Compe-tence8 in 2013 that called its members to “align medical education, training and assessment with the competency framework to help physicians demonstrate mastery and excellence throughout their careers.” The Coalition solicited the input of the public (nonphysician) members of its 12 member organizations regarding physician competencies and competency domains. This consensus was reviewed by the Panel in its deliberations.
One of the primary accomplishments of the Panel was to recommend a revised COMLEX-USA blueprint con-sisting of 2 dimensions—Competency Domains and Clinical Presentations—updated from the current dimen-sions of Patient Presentations and Physician Tasks. As a high-stakes summative examination, COMLEX-USA follows NBOME’s Standards for Quality Assurance9 that are aligned with the aforementioned Standards, which state that validity, the fundamental consideration of a test, is “…the degree to which all the accumulated evi-dence supports the intended interpretation of test scores for the proposed use.”7(p14) As a result, a major focus of the Panel’s work in developing the new blueprint was gath-ering and integrating multiple sources of evidence based on established standards and psychometric principles. COMLEX-USA and the United States Medical
Li-censing Examination as valid and reliable for their in-tended and respective purposes and concluded that the validity of COMLEX-USA is exemplary. A summary of published evidence for the validity of the current COMLEX-USA series for licensure and other secondary uses can be found on the National Board of Osteopathic Medical Examiners (NBOME) website.5
The purpose of an examination blueprint is to outline the content and format specifications for the examina-tion, including the percentage of total test questions and other content associated with each topic.6 In the years since its implementation, the COMLEX-USA blueprint has been reviewed and revised regularly to reflect the practice of osteopathic medicine and to be consistent with the recommendations of the Standards for Educational and Psychological Testing (ie, Stan-dards) established by the American Educational Re-search Association, the American Psychological Association, and the National Council on Measurement in Education in 2014.7 The addition of a performance assessment of fundamental clinical skills, COMLEX-USA Level 2-Performance Evaluation, was added to the series in 2004. Additional test item formats have been developed and incorporated into the examinations. In 2018-2019, the NBOME plans to implement the en-hanced test blueprint and specifications described herein.
Blueprint Development
To ensure that COMLEX-USA is most effectively posi-tioned to anticipate new trends and changes in osteo-pathic medical practice and education, the NBOME commissioned the Blue Ribbon Panel (ie, Panel) in 2010. The Panel was charged with outlining a plan to imple-ment a 2-decision–point, competency-based COMLEX-USA that is consistent with the NBOME’s mission to protect the public by providing the means to assess com-petencies for osteopathic medicine and related health care professions. Decision Point One was defined as the grouping of predoctoral assessments leading to suc-cessful promotion along the licensure pathway for entry
MEDICAL
EDUCA
These Standards describe the importance of examina-tion blueprints being job-related and incorporating the knowledge, skills, and abilities required for effective practice for the profession. The requirement of being job-related is met in part by conducting a practice anal-ysis, which is “the systematic study of a profession un-dertaken to identify and describe the job responsibilities of those employed in the profession.”10(p29) Practice analyses may include a variety of procedures, including a comprehensive review of relevant research, such as national survey data and other relevant literature; collec-tion and evaluacollec-tion of expert opinion in that profession; and completion of surveys, such as task inventories de-signed to elicit information from stakeholders.11 The Panel’s initial step was to focus on the compe-tency domains, with a starting point of previous NBOME research that defined the osteopathic medical competency domains and described them primarily from the licensure assessment perspective.12,13 To sup-plement these data, the Panel conducted a literature survey that included review of national and interna-tional medical competency models that have influenced the expectations of good medical practice. Among these frameworks were AACOM’s Osteopathic Core Compe-tencies for Medical Students,14 CanMEDS 2005 Physi-cian Competency Framework,15 and the ACGME Outcomes Project,16 as well as current research on physician revalidation.17 The result of this research was the updated Fundamental Osteopathic Medical Compe-tency Domains (FOMCD) 2011,18 which delineated 7 competency domains comprising required elements and measurable outcomes.
A second outcome of the competency research con-ducted by the Panel was the publication of FOMCD 2016,19 revised from FOMCD 2011 to provide a clarified presentation of required elements and outcomes that comprise the competency domains. The required ele-ments detail the essential foundational specifications, including specific, definable knowledge, skills, experi-ences, attitudes, values, or behaviors that make up the standards for the competency domain. Each required ele-ment includes 1 or more outcomes and more explicit
description statements of desired abilities, which are further classified as anticipated to be measured, attested, or not measured in the COMLEX-USA series.
To obtain current data regarding the clinical prac-tices of osteopathic physicians (ie, DOs), the Panel studied data from the 2010 National Ambulatory Medical Care Survey20 that reports medical care statis-tics related to DOs’ practice, such as patient demo-graphics and visit characteristics including the principal reason for visit. Comparing data from DOs and allopathic physicians (ie, MDs) confirmed that DOs encounter more patients with musculoskeletal (13.4% DOs vs 9.2% MDs) and respiratory (13.5% DOs vs 9.9% MDs) complaints than MDs.
The Panel also evaluated data from the 2010 National Hospital Ambulatory Medical Care Survey,21 the 2010 National Hospital Discharge Survey,22 and the 2004 Na-tional Nursing Home Survey.23 These analyses demon-strated some of the distinctive features of osteopathic medical practice and confirm that a higher percentage of patients with musculoskeletal and respiratory system complaints present to DOs than to MDs.24 This finding is reflected in the blueprint weighting.
Panel Research and Surveys
In addition to reviewing current research and national survey data, the Panel sponsored research projects that contributed to the practice analysis, including a survey of the clinical procedures taught during residency. Other research efforts investigated the relationship between competency ratings of residents and COMLEX-USA scores,25,26 contributing convergent evidence of validity.7 With the intention of adding broad stakeholder opin-ions to the practice analysis, the Panel sponsored a test development survey that was distributed to more than 15,000 persons in the osteopathic medical profession, including faculty and deans of colleges of osteopathic medicine, the Association of Osteopathic Directors and Medical Educators, the American Association of Osteo-pathic Examiners, and osteoOsteo-pathic medical students from the graduating classes of 2009 to 2013.
MEDICAL
EDUCA
topics that should be retained as is, retained but modi-fied, eliminated, or added. By presenting the results of the survey, members of the workgroup used data to achieve consensus and arrive at a focused direction for both modernizing the language and reorganizing the categories to propose the new Clinical Presentations dimension. The Blueprint Workgroup continued their research by evaluating national survey data.20-23 To supplement these, they reviewed DO and MD medical school curricular programs in the United States, inter-national medical school curricular programs, graduate medical education objectives, and competencies and milestones to arrive at consensus regarding the evolving best practices and standard content specifications across the educational spectrum. Using their clinical expertise and a consensus-driven process focusing on how patients present to DOs, the Blueprint Workgroup further defined the topics and subtopics that comprised the categories of the Clinical Presentations dimension. These recommendations were presented back to the Panel for their consideration.
The 2018-2019
Master Blueprint
The outcome of the research and analysis conducted by the Panel and the Blueprint Workgroup was the devel-opment of an enhanced, competency-based, 2-dimen-sional blueprint that is aligned with the evolving practice of osteopathic medicine. It includes as its intro-ductory preamble the tenets of osteopathic medicine27:
■ The body is a unit; the person is a unit of body, mind, and spirit.
■ The body is capable of self-regulation, self-healing, and health maintenance.
■ Structure and function are reciprocally interrelated.
■ Rational treatment is based upon an understanding of the basic principles of body unity, self-regulation, and the interrelationship of structure and function. This survey asked respondents to rate the
impor-tance of the specific competency domain required ele-ments at Decision Point One (ie, entry into supervised practice, residency training) and Decision Point Two (ie, entry into unsupervised practice, unsupervised practice/licensure). These domains included the 7 Competency Domains: Osteopathic Principles, Prac-tice, and Manipulative Treatment; Osteopathic Patient Care and Procedural Skills; Application of Knowledge for Osteopathic Medical Practice; Practice-Based Learning and Improvement in Osteopathic Medical Practice; Interpersonal and Communication Skills in the Practice of Osteopathic Medicine; Professionalism in the Practice of Osteopathic Medicine; and Systems-Based Practice in Osteopathic Medicine. The elements receiving the highest averages among DO respondents for Decision Points One and Two are listed in Table 1. Information from the survey was evaluated by the Panel and considered in the final determination of competency-based content for the enhanced blueprint.
The Blueprint Workgroup
As this research was being conducted, the Panel estab-lished the Blueprint Workgroup with members from rep-resentative specialties from the osteopathic medical profession who were tasked to focus on additional practice analysis activities. The Blueprint Workgroup was charged with developing recommendations to the Panel based on empirical research and their expert opinion for the devel-opment of the blueprint, including its nomenclature and weighting, particularly as related to the Clinical Presenta-tions dimension. A clinical presentation was defined as the manner in which a particular patient, group of patients, or community presents to DOs. Clinical presentations are high-frequency, high-impact categories based on evidence from osteopathic medical practice and are further classi-fied as categories and topics.
Using the current blueprint’s Patient Presentations dimension as a starting point for their research, mem-bers of the Blueprint Workgroup completed the Blue-print Workgroup Survey that asked them to identify
MEDICAL
EDUCA
Required Element Average Rating
Competency Domains: Decision Point One
Commitment to continuous learning 4.51
Ability to work effectively with other members 4.40 of the interprofessional collaborative team
Ability to gather data from all sources, including 4.37 patients’ secondary sources, records, and
physical examination results
Ability to formulate a prioritized differential diagnosis 4.33
Humanistic behavior 4.32
Rapport building with patients 4.27
Effective written and electronic communication 4.27 Ability to give information to patients clearly, 4.26 including collaboration on treatment, informed
consent, and lifestyle modification
Ability to counsel and educate patients consistent 4.18 with their ability to understand
Treatment of patients using best medical evidence 4.16 Ability to develop a safe, evidence-based, 3.84 cost-effective, patient-centered care plan
Ability to perform basic clinical procedures essential 3.77 for a general osteopathic medical practice
Clinical Presentations: Decision Point Two
Ability to formulate a prioritized differential diagnosis 4.78 Ability to gather data from all sources, including 4.70 patients’ secondary sources, records, and
physical examination results
Ability to counsel and educate patients consistent 4.68 with their ability to understand
Ability to give information to patients clearly, 4.68 including collaboration on treatment, informed
consent, and lifestyle modification
Rapport building with patients 4.66
Ability to develop a safe, evidence-based, 4.63 cost-effective, patient-centered care plan
Treatment of patients using best medical evidence 4.60
Commitment to continuous learning 4.59
Ability to work effectively with other members 4.58 of the interprofessional team
Effective written and electronic communication 4.56
Humanistic behavior 4.45
Ability to perform basic clinical procedures essential 4.37 for a general osteopathic medical practice
a Items were rated on a scale of 1 (“not important at all”) to 5 (“extremely important”).
MEDICAL
EDUCA
cautioned residency program directors to avoid the overuse or sole use of any examination program for screening or hiring of residents, the evidence supporting these secondary uses is relevant. The ACGME recog-nizes and supports the use of COMLEX-USA for DOs in its programs,5 stating that “COMLEX and USMLE [United States Medical Licensing Examination] are both acceptable to the ACGME….We recognize the important role that COMLEX-USA plays in quality for osteopathic medical education and training.”28
Next Steps
In developing the Master Blueprint 2018-2019, the Panel sponsored new research, evaluated existing research, considered expert opinion, and relied on psychometric best practices to create an evidence-based blueprint de-sign for the COMLEX-USA series. Because the updated blueprint calls for new content with different weighting, its implementation may result in changes to the content of other assessments currently being administered in os-teopathic medical education. With this in mind, the NBOME has begun communicating the Master Blue-print 2018-2019 to educators.29
The new blueprint will be implemented beginning with Decision Point Two (COMLEX-USA Level 3) in 2018. With the goal of assessing across a broader compe-tency subset, COMLEX-USA Level 3 will be extended from 1 day to 2 days and will include multiple-choice items as well as novel item formats. The new blueprint will be implemented for Decision Point One examina-tions (COMLEX-USA Level 1, Level 2-Cognitive Evaluation, and Level 2-Performance Evaluation) begin-ning in 2019. These examinations will remain similar in format but will follow the new blueprint’s content out-line. Test specifications for each level and component of COMLEX-USA will be available publicly in 2017 to provide more detailed direction to stakeholders. Updated format specifications are currently available on the NBOME’s website.30
In keeping with its focus on maximizing the validity evidence for COMLEX-USA, the NBOME continues to The Competency Domains dimension (Table 2) in the
new blueprint was defined as “related sets of founda-tional abilities and represent the required elements and outcomes that define the knowledge, skills, experience, attitudes, values, and behaviors of established profes-sional standards”19 and provide an overall framework for the practice of osteopathic medicine.
The Competency Domains dimension of the blue-print includes the measured outcomes of FOMCD 2016. Measured outcomes are considered statements of well-defined abilities, including knowledge, skills, experi-ences, attitudes, values, or behaviors that are observable and measurable and can be directly assessed in a reliable manner in the assessments that make up the COMLEX-USA series. Although attested outcomes and not-mea-sured outcomes, which typically would require workplace-based assessment, are not currently expected to be assessed in the new COMLEX-USA, these out-comes could provide opportunities for further collabora-tion and research and eventually be included in an expanded portfolio for licensure. The Clinical Presenta-tions dimension of the new blueprint defines the manner in which a particular patient, group of patients, or com-munity presents to DOs (Table 2).
Ultimately, the weighting for competency domains and clinical presentations, based on the comprehensive analyses of osteopathic medical practice, was determined by the Panel.5
Primary and Secondary
Uses of COMLEX-USA
The Panel affirmed that the primary and intended pur-pose of COMLEX-USA is for the licensure of DOs. However, the Panel acknowledged the evidence for im-portant secondary uses of the examination results, in-cluding screening applicants for residency programs, summative evaluation for graduation from a college of osteopathic medicine, and promotion and academic progress evaluation in colleges or within residency pro-grams. Although the NBOME supports holistic admis-sion reviews in the residency application process and has
MEDICAL
EDUCA
During the past 5 to 7 years, there has been a focus on “milestones,”33 the developmental landmarks that mark the stepwise progression of a resident toward competency development acquired progressively during his or her residency training. As conceptualized by the ACGME, the milestones are delineated by spe-cialty for each competency in terms of 5 levels of profi-ciency, from novice to expert, and require performance judgments by trained observers in the workplace. With the advent of the single accreditation system, the col-laboration of the ACGME, AOA, and AACOM has re-sulted in the creation of osteopathic milestones for residency programs that apply for “osteopathic recogni-tion” status within the ACGME.34 These osteopathic develop novel item formats for use on COMLEX-USA
to assess a wider range of competencies and to increase the authenticity of its assessments. Among them are ex-panded use of multimedia test items, items that incorpo-rate patients’ heart and lung sounds, and the Key Features/Clinical Decision Making31 format that was in-troduced in COMLEX-USA Level 3 in 2014 and is cur-rently in use on other medical licensing examinations, notably by the Medical Council of Canada.32 Currently, the NBOME is investigating the potential of point-of-care item formats that would assess candidates’ use of resources such as clinical decision-making tools and re-source applications typically available to physicians in actual clinical practice.
Table 2.
Weighting for COMLEX-USA Competency Domains and Clinical Presentations
Dimensions Minimum Weighting, %
Competency Domains: Dimension One
Osteopathic Principles, Practice, and Manipulative Treatment 10
Osteopathic Patient Care and Procedural Skills 25
Application of Knowledge for Osteopathic Medical Practice 30 Practice-Based Learning and Improvement in Osteopathic Medical Practice 5 Interpersonal and Communication Skills in the Practice of Osteopathic Medicine 10
Professionalism in the Practice of Osteopathic Medicine 5
Systems-Based Practice in Osteopathic Medicine 5
Clinical Presentations: Dimension Two
Community Health and Presentations Related to Wellness 12
Patient Presentations Related to:
Human Development, Reproduction, and Sexuality 5
Endocrine System and Metabolism 5
Nervous System and Mental Health 10
Musculoskeletal System 13
Genitourinary System 5
Gastrointestinal System and Nutritional Health 10
Circulatory and Hematologic Systems 10
Respiratory System 10
Integumentary System 5
Abbreviation: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination-USA. Source: Reprinted with permission from the National Board of Osteopathic Medical Examiners.
MEDICAL
EDUCA
Acknowledgments
We thank Melissa Turner, MS, and Laura Barrett from the Strategy, Quality and Communications Department at the National Board of Osteopathic Medical Examiners, for their review and editorial assistance.
Author Contributions
Drs Gimpel, Horber, and Sandella provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; Drs Knebl and Thornburg drafted the article or revised it critically for important intellectual content; Dr Gimpel gave final approval of the version of the article to be published; and Drs Gimpel, Horber, and Sandella agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. References
1. Sandella JM, Gimpel JR, Smith LL, Boulet JR. The use of COMLEX-USA and USMLE for residency applicant selection. J Grad Med Educ. 2016:8(3):358-363. doi:10.4300/JGME-D-15-00246.1
2. Osborn GG, Meoli FG, Buser BR, Clearfield MB, Bruno JP, Sumner-Truax L. The Comprehensive Osteopathic Medical Licensing Examination, COMLEX-USA: a new paradigm in testing and evaluation. J Am Osteopath Assoc. 2000;100(2):105-111. 3. Blueprint Project Team. Blueprint Project—Qualifying
Examinations Blueprint and Content Specifications. Ottawa, Ontario: Medical Council of Canada; 2014. http://mcc.ca/wp-content/uploads/Blueprint-Report.pdf. Accessed February 9, 2017.
4. Step 1. United States Medical Licensing Examination website. http://www.usmle.org/step-1/. Accessed February 9, 2017. 5. National Board of Osteopathic Medical Examiners.
Residency program director’s guide to COMLEX-USA. https://www.nbome.org/docs/RPD.pdf.
Accessed February 9, 2017.
6. Downing SM. Twelve steps for effective test development. In: Downing SM, Haladyna TM, eds. Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum; 2016. 7. Standards for Educational and Psychological Testing.
2nd ed. Washington, DC: American Educational Research Association; 2014.
8. Coalition for Physician Accountability. Consensus statement on a framework for professional competence by the Coalition for Physician Accountability. http://www.physicianaccountability.org/pdf/Coalition _Competencies_Consensus_Statement.pdf. Accessed February 9, 2017.
9. National Board of Osteopathic Medical Examiners. Standards for Quality Assurance. Chicago, IL: National Board of Osteopathic Medical Examiners; 2015. 10. Raymond M. Developing and Administering Practice
Analysis Questionnaire [module]. Philadelphia, PA: National Council on Measurement in Education; 2005. milestones incorporate osteopathic distinctiveness and
competency-based assessment frameworks outlined for DOs in residency training.
The ACGME’s milestones are intended for use in the formative assessment of learner development, and the NBOME’s competency domains with their required ele-ments and outcomes are considered the behaviors, skills, values, and attitudes that all DOs are expected to demon-strate, and as such are for use as summative assessment. Based on work undertaken by AACOM,14 some align-ment is possible between the competency domains, re-quired elements and outcomes, Entrustable Professional Activities, and other formative assessments. As under-scored in FOMCD 2016, the NBOME has attempted to harmonize the language and descriptions of competency domains, competencies, milestones, and Entrustable Professional Activities to facilitate alignment among learners both in undergraduate and graduate medical ed-ucation. Despite these overall differences, the NBOME is seeking future opportunities for harmonization with the expectation that assessment of some attested or not-measured outcomes may someday contribute valuable information to physicians and trainees, state licensing boards, deans, and program directors.
Conclusion
Now under the governance of NBOME’s COMLEX-USA Composite Examination Committee and its Board of Directors, final work on score reporting and continu-ance of systematic standard-setting processes are un-derway. The NBOME continues to study ways to limit construct-irrelevant variance in COMLEX-USA test scores, with appropriate attention to fairness and the paramount importance of protecting the public and en-hancing the quality of patient care and public health. In this regard, the NBOME recognizes the educational ef-fect that the COMLEX-USA series has in motivating students and residents to prepare in a manner that has educational benefit, as well as the catalytic effect such assessment can have in driving future learning and im-proving the care our patients receive.
MEDICAL
EDUCA
Services, Centers for Disease Control and Prevention; 2010. http://www.cdc.gov/nchs/data/ahcd/nhamcs_emergency/2010 _ed_web_tables.pdf. Accessed February 16, 2017. 22. National hospital discharge survey: data highlights - selected
tables. Centers for Disease Control and Prevention website. http://www.cdc.gov/nchs/nhds/nhds_tables.htm. Accessed February 16, 2017.
23. Jones AL, Dwyer LL, Bercovitz AR, Strahan GW. The National Nursing Home Survey: 2004 overview. Vital Health Stat. 2009;13(167):1-155.
24. Boulet JR, Gimpel JR, Errichetti AM, Meoli FG. Using National Medical Care Survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians. J Am Osteopath Assoc. 2003;103(5):225-231. 25. Langenau EE, Pugliano G, Roberts WL. Relationships between
high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents. Med Educ Online. 2011;16:7362. doi:10.3402/meo.v16i0.7362 26. Langenau EE, Pugliano G, Roberts WL, Hostoffer R.
Summary of ACOP (American College of Osteopathic Pediatricians) program directors’ annual reports for first-year residents and relationships between resident competency performance ratings and COMLEX-USA test scores. Electronic J Am Coll Osteopath Physicians. 2010;2(7). 27. Tenets of osteopathic medicine. American Osteopathic
Association website. https://osteopathic.org/inside-aoa /about/leadership/Pages/tenets-of-osteopathic-medicine.aspx. Accessed February 16, 2017.
28. Joint AACOM & AODME 2015 conference program. American Association of Colleges of Osteopathic Medicine website. http://www.aacom.org/news-and-events/conference /past-conferences/2015-AC/program/wednesday. Accessed February 16, 2017.
29. COMLEX-USA master blueprint. National Board of Osteopathic Medical Examiners. https://www.nbome.org/docs/COMLEX_ Master_Blueprint_2018-2019.pdf. Accessed February 16, 2017. 30. Level 1, level 2, and level 3 computer-based cognitive
examinations. National Board of Osteopathic Medical Examiners website. http://www.nbome.org/comlex-cbt.asp#a1. Accessed March 1, 2017.
31. Farmer EA, Page G. A practical guide to assessing clinical decision-making skills using the key features approach. Med Educ. 2005;39(12):1188-1194.
32. Medical Council of Canada. Guidelines for the Development of Key Features Problems & Test Cases. Ottawa, Ontario: Medical Council of Canada; 2012. http://mcc.ca/wp-content /uploads/CDM-Guidelines.pdf. Accessed February 16, 2017. 33. Milestones. Accreditation Council for Graduate Medical Education
website. http://www.acgme.org/what-we-Do/Accreditation /Milestones/Overview. Accessed February 16, 2017. 34. Accreditation Council for Graduate Medical Education and the
American Osteopathic Association. The Osteopathic Recognition Milestone Project. Chicago, IL: Accreditation Council for Graduate Medical Education; 2015. https://www.acgme.org/Portals/0/PDFs /Milestones/OsteopathicRecognitionMilestones.pdf.
Accessed February 16, 2017. © 2017 American Osteopathic Association 11. Knapp J, Knapp L. Practice analysis: building the foundations
for validity. In: Impara JC, ed. Licensure Testing: Purposes, Procedures, and Practices. Lincoln, NE: Buros Institute of Mental Measurement; 1995:91-116.
12. National Board of Osteopathic Medical Examiners.
The Seven Osteopathic Medical Competencies: Considerations for Future Testing and the Practice of Osteopathic Medicine. Chicago, IL: National Board of Osteopathic Medical Examiners; 2006. https://license.k3systems.com/LicensingPublic/docs /OsteopathicCompetenciesSeptember2006.pdf. Accessed February 16, 2017.
13. National Board of Osteopathic Medical Examiners. The Fundamental Osteopathic Medical Competencies: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Chicago IL: National Board of Osteopathic Medical Examiners; 2009. https://www.aacom .org/docs/default-source/core-competencies/nbome6core -comp2009.pdf?sfvrsn=4. Accessed February 16, 2017. 14. Osteopathic Considerations for Core Entrustable Professional
Activities (EPAs) for Entering Residency. Chevy Chase, MD: American Association of Colleges of Osteopathic Medicine; 2016. https://www.aacom.org/docs/default-source/med-ed-presentations/ core-epas.pdf?sfvrsn=10. Accessed February 16, 2017. 15. The Royal College of Physicians and Surgeons of Canada.
The CanMEDS 2005 Physician Competency Framework. Ottawa, Ontario: The Royal College of Physicians and Surgeons of Canada; 2005. http://www.ub.edu/medicina_ unitateducaciomedica/documentos/CanMeds.pdf. Accessed February 16, 2017.
16. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29(7):648-654.
doi:10.1080/01421590701392903
17. Medical Council of Canada. Recalibrating for the 21st Century: Report of the Assessment Review Task Force of the Medical Council of Canada. Ottawa, Ontario: Medical Council of Canada; 2011. http://mcc.ca/wp-content/uploads/Reports-assessment-review-task-force.pdf. Accessed February 16, 2017. 18. National Board of Osteopathic Medical Examiners.
The Fundamental Osteopathic Medical Competency Domains: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Chicago, IL: National Board of Osteopathic Medical Examiners; 2011. https://www.nbome.org /docs/NBOME%20Fundamental%20Osteopathic%20Medical %20Competencies.pdf. Accessed February 16, 2017. 19. National Board of Osteopathic Medical Examiners.
The Fundamental Osteopathic Medical Competency Domains: Guidelines for Osteopathic Medical Licensure and the Practice of Osteopathic Medicine. Chicago, IL: National Board of Osteopathic Medical Examiners; 2016. http://www.nbome.org/docs/Flipbooks /FOMCD/index.html#p=1. Accessed February 16, 2017. 20. Centers for Disease Control and Prevention. National Ambulatory
Medical Care Survey: 2010 Summary Tables. Atlanta, GA: US Dept of Health and Human Services, Centers for Disease Control and Prevention; 2010. https://www.cdc.gov/nchs/data /ahcd/namcs_summary/2010_namcs_web_tables.pdf. Accessed February 16, 2017.
21. Centers for Disease Control and Prevention. National Hospital Ambulatory Medical Care Survey: 2010 Emergency Department Summary Tables. Atlanta, GA: US Dept of Health and Human
