Title: Approaches for evaluating learning outcomes in competency-based educational programs: a protocol for a scoping review
Authors: Carrie K. McMullen1, Katie Clow1, and Jane Parmley1
1Department of Population Medicine, Ontario Veterinary College, University of Guelph, Guelph, ON, Canada, N1G 2W1.
Author contributions: CKM drafted the review protocol with input from KC and JP.
Registration: This protocol is archived in the University of Guelph’s institutional repository (The Atrium) available at: https://atrium.lib.uoguelph.ca/xmlui/handle/10214/10046. The subsequent review will be reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR: http://www.prisma-statement.org/Extensions/ScopingReviews) guidelines (Tricco et al., 2018) and the methodological framework proposed by Arksey and O’Malley (2005).
Amendments: Any amendments made to this protocol following its registration will be documented in the final manuscript as protocol deviations.
Support: Funding for this project has been made possible through a contribution from the Public Health Agency of Canada.
INTRODUCTION Rationale:
Core competencies such as health knowledge, issues affecting humans, animals, plants, and the environment, principles of research, and program/policy evaluation are recommended for One Health programs in North America (Togami et al., 2018). Identifying appropriate methods to evaluate learning outcomes will indicate whether students are able to demonstrate the One Health core competencies that will equip them with the necessary skills to tackle complex challenges in Canada, such as climate change and infectious diseases. A scoping review of available methodologies for evaluating learning outcomes in competency-based educational programs will enable researchers at the University of Guelph to determine if a new framework should be drafted for evaluating One Health learning outcomes within the university, or whether existing frameworks can be tailored to best suit evaluation of the One Health learning outcomes at the University of Guelph, and ultimately in One Health programs at universities across Canada. Objectives: The objective of this protocol is to define the methods for a scoping review that will characterize available literature on methods used to evaluate learning outcomes in competency-based educational programs at the post-secondary level.
Research questions: The specific review questions to be addressed in this protocol, and the subsequent scoping review, are as follows: What methods exist for evaluating competency-based learning outcomes? Can we adapt or adopt these methods for use in current One Health programs?
ELIGIBILITY CRITERIA
Participants: Participant details are not relevant for study selection in this scoping review concept. Concept: The concept characterized in this scoping review is methodologies or approaches to evaluating learning outcomes at the post-secondary education level (i.e., diploma, undergraduate, graduate, post-graduate), which also includes professional programs such as the Doctor of Medicine and Doctor of Veterinary Medicine programs. Evaluation methods for learning outcomes can be at the course, program, departmental, or institutional level. Therefore, a range of possible evaluation methodologies will be eligible for inclusion, such as student-centered evaluations, curriculum-centered evaluations, or faculty-centered evaluations.
Context: This scoping review will consider articles that evaluate learning outcomes that are defined by competencies, and therefore we will consider competency-based educational programs at the post-secondary level. However, this review will not be limited to a specific discipline. Post-secondary education programs or courses eligible for inclusion are diploma, undergraduate, graduate, or post-graduate and must be from a university or college. Professional designation programs such as, but not limited to, Doctor of Medicine, Medical Residencies, Doctor of Veterinary Medicine, Surgical Residencies, Law, and Master of Business Administration will also be eligible for inclusion. Professional programs focused on adult-continuing education or career development will not be eligible for inclusion in this scoping review (e.g., Chartered Professional Accounting, Infection Prevention and Control).
Publication Types: The following publication types and report characteristics are eligible for inclusion in this scoping review: primary research articles, conference proceedings (provided they are ≥500 words), dissertations and theses that report primary data, and grey literature if primary data is reported. The following publication types are ineligible for inclusion in this scoping review: narrative reviews, systematic reviews, critical reviews, letters to the editor, textbooks, and opinion articles. However, if a review has been conducted on a similar research topic to this scoping review we will check the bibliography for additional records.
Additionally, other eligibility criteria that will be included are as follows: Publication date: There will be no publication date limit.
Report characteristics: The study must be published in English and the publication can be both published and non-published if the article is ≥500 words and reports on primary data. Articles will not be included beyond full-text screening if the full-text cannot be located using library resources at the University of Guelph.
Information Sources: The search will include a range of biologically relevant databases to identify primary literature. Table 1 represents the databases to be searched.
Table 1 Databases to be searched for relevant literature
Database Interface/URL
Ovid MEDLINE(R) and Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Daily and Versions(R)
Ovid SP
CAB Direct CAB Interface
Academic Search Premier EBSCOhost
Educational Resources Information Centre (ERIC)
ProQuest
Search strategy: The search strategy used for this review was developed using key concept terms and words. Concept and context terms will be connected using Boolean operators ‘AND’ or ‘OR’. Table 2 and Table 3 represent the search strings that will be used for this review as applied in CABI and MEDLINE databases.
Table 2 Results of a search to identify articles reporting an evaluation framework for learning outcomes in competency-based education conducted in CAB Direct (via CABI) on January 11, 2021
# Search Terms Results
1 competenc* OR “core competenc*” OR “competency-based” OR “competency based” OR “outcome-based” OR “outcome based” OR “standards-based” OR “standards based”
29 425
2 “competency based education” 185
3 1 OR 2 29 425
4 “learning outcome*” OR “learning objective*” 1010
5 3 OR 4 30 277
6 education OR curricul* 374 411
7 5 AND 6 6103
8 undergrad* OR grad* OR university OR college OR “post-secondary” OR postsecondary
5 362 426
9 7 AND 8 4762
10 evaluat* OR assess* OR analyz* 3 001 413
11 9 AND 10 2396
Table 3 Results of a search to identify articles reporting an evaluation framework for learning outcomes in competency-based education conducted in Ovid MEDLINE(R) and Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Daily and Versions(R) 1946 to January 8, 2021, conducted on January 11, 2021. Search term #2 (focus) and #6 (focus) are subject heading search results, the remaining are key word searches
# Search Terms Results
1 (competenc* or core competenc* or competency-based or competency based or outcome-based or outcome based or standards-based or standards based).mp. [mp=title, abstract, original title, name of substance word,
subject heading word, floating sub-heading word, keyword heading word, organism supplementary concept word, protocol supplementary concept word, rare disease supplementary concept word, unique identifier, synonyms]
2 *Competency-Based Education/ 2484
3 1 OR 2 203 085
4 (learning outcome* or learning objective*).mp. 8566
5 3 OR 4 209 499
6 *Education/ OR *Curriculum/ 33 832
7 5 AND 6 6616
8 (undergrad* or grad* or university or college or post-secondary or postsecondary).mp.
1 456 118
9 7 AND 8 3594
10 (evaluat* or assess* or analyz*).mp. 7 464 837
11 9 AND 10 2284
In addition, one reviewer will hand-search the following websites for relevant grey literature publications:
• National Institute for Learning Outcome Assessments (NILOA) • Higher Education Quality Council of Ontario
• Association of American Colleges and Universities
• Organisation for Economic Co-operation and Development (OECD)
In order to validate the search strategy, 15 relevant journal articles were pre-selected by CKM and checked for inclusion in the search (Table 4).
Table 4 Article title/author information for 15 relevant journal articles pre-selected by CKM to check for inclusion in the search in order to validate the search strategy
Author (Year)
Article Title Country Program
Discipline Program Level 1 Berns (2010) A survey-based evaluation of self-perceived competency after nephrology fellowship training USA Human Medicine Professional (fellowship) 2 Berrian et al. (2020)
Developing a global One Health workforce: the “Rx One Health Summer Institute” approach
East Africa
One Health Professional
3 Britten et al. (2014)
Using core competencies to build an evaluative
framework: outcome
assessment of the University of Guelph Master of Public Health program
Canada Public Health
4 Gouvea et al. (2013)
A framework for analyzing interdisciplinary tasks: implications for student learning and curricular design
USA Physics Undergraduate
5 Hamilton et al. (2020)
Mapping veterinary curricula to enhance World
Organisation for Animal Health (OIE) Day 1 competence of veterinary graduates
USA Veterinary Professional
6 Hodgson et al. (2013)
Beyond NAVMEC:
Competency-based veterinary education and assessment of the professional competencies
USA Veterinary Professional
7 Joyner (2016)
Curriculum mapping: a method to assess and refine undergraduate degree programs USA Food Science Undergraduate 8 Lam and Tsui (2016) Curriculum mapping as deliberation – examining the alignment of subject learning outcomes and course
curricula Hong Kong Education Undergraduate, Post-graduate 9 Larsen and Reif (2019)
Leveling the core
competencies of public health nursing to evaluate senior baccalaureate nursing students
USA Nursing Undergraduate
10 Leibbrandt et al. (2005)
National comparative curriculum evaluation of baccalaureate nursing
degrees: a framework for the practice based professions
Australia Nursing Undergraduate
11 Luvira et al. (2018)
Formative evaluation and learning achievement in epidemiology for preclinical medical students India Human Medicine Professional 12 Negandhi et al. (2015) Developing core
competencies for monitoring and evaluation tracks in South Asian MPH programs
14 Smith and Dollase (1999)
AMEE guide No. 14: Outcome-based education: Part 2 – Planning,
implementing and evaluating a competency-based curriculum USA Human Medicine Professional 15 Woeber (2018) Development and implementation of a competency-based clinical evaluation tool for midwifery education
USA Nursing Professional
STUDY RECORDS
Data management: The results from the searches will be downloaded into a bibliographic software program (EndNote X7, Clarivate Analytics, Philadelphia) and duplicated. The de-duplicated references from the search will be loaded into an online reference management software (DistillerSR®, Ottawa, ON, Canada), and further duplicated using this software’s de-duplication process. All study selection and characterization steps will also be conducted in DistillerSR® using forms specifically created for this project. All reviewers involved in the screening and data extraction process will have had previous training in relevant pedagogical principles and scoping review methods. Prior to both stages of screening, a pilot test will be conducted to ensure reviewer compatibility and consistent data collection using the forms created in DistillerSR®.
Selection process: This review will entail two stages of screening. First, the titles and abstracts of each article identified by the search strategy will be assessed for eligibility. The pilot test for title and abstract screening will involve all reviewers screening 150 articles for eligibility, followed by a consensus meeting. Any disagreements will be thoroughly discussed to ensure all reviewers are consistent for the remaining articles. The remaining titles and abstracts will be independently assessed for relevance using the following primary screening questions by two independent reviewers:
1) Is a university or college educational program discussed in the title/abstract? YES (neutral), NO (exclude), UNCLEAR (neutral)
2) Are learning outcomes or competencies discussed in the title/abstract? YES (neutral), NO (exclude), UNCLEAR (neutral)
3) Are methods for evaluating learning outcomes or competency described in the title/abstract?
YES (neutral), NO (exclude), UNCLEAR (neutral)
4) Is this a primary research article? (i.e., peer-reviewed article, conference proceeding (≥500 words), dissertation or thesis that reports primary data, or grey literature that reports primary data)
YES (neutral), NO (exclude), UNCLEAR (neutral) 5) Is the title/abstract available in English?
A reference will only be excluded if both reviewers give a final decision to exclude (i.e., both reviewers agree the response to any of the above questions is NO). Any conflicts will be resolved by consensus, and if agreement cannot be reached, a third member of the review team will be consulted.
The second stage of screening will involve assessing the full-text of each article for eligibility by two independent reviewers. The full-text articles of the citations deemed eligible by the first stage of screening will be retrieved and uploaded into DistillerSR®. If a full-text article cannot be retrieved online using available University of Guelph resources, articles will be requested via a University of Guelph librarian. All reviewers will conduct the pilot test involving 25 full-text articles to ensure reviewer consistency in applying questions. The following secondary screening questions will be used to assess the full-text of each article by two reviewers working independently:
1) Is the study available in English? YES (neutral), NO (exclude)
6) Is this a primary research article? (i.e., peer-reviewed article, conference proceeding (≥500 words), dissertation or thesis that reports primary data, or grey literature that reports primary data)
YES (neutral), NO (exclude)
2) Is a university or college educational program described in the full-text article? YES (neutral), NO (exclude)
3) Is the educational program competency-based? YES (neutral), NO (exclude)
4) Are learning outcomes or competencies the discussed in the full-text? YES (neutral), NO (exclude)
5) Are methods for evaluating learning outcomes or competencies proposed in the full-text? YES (neutral), NO (exclude)
A reference will only be excluded if both reviewers answer NO to any screening question. Any conflicts will be resolved by consensus. If consensus cannot be reached, a third person on the review team will be consulted. Reasons for exclusions at this stage of screening will be documented.
Data extraction process: Two reviewers will extract data from eligible studies independently using a structure pre-tested form created in DistillerSR®. This form will be pilot tested using 10 references by all reviewers in order to ensure consistency in data extraction. Authors will not be contacted to request missing data or to clarify published results.
Data items: The following information will be extracted:
A) General study characteristics: year of publication, year study was conducted, country where study was conducted (if not reported, use location of first author)
C) Program level: diploma, undergraduate program, graduate program, post-graduate program, professional program (includes Doctor of Medicine program, Doctor of Veterinary Medicine program, Dental program, etc.) (checkbox)
D) Program discipline: nursing, medicine, veterinary, dental, engineering, business, etc. (checkbox)
E) Competency Description:
a. What competencies were assessed within the paper? (checkbox) b. Were the competencies directly assessed?
c. Were learning outcomes used as a proxy for assessing competency? F) Evaluation Methodology:
a. Author-derived method b. Adapted method
c. Previously published method d. Type of evaluation:
i. Student-oriented (survey/focus group of student perceptions, student graded assessments, etc.)
ii. Curriculum-oriented (e.g., curriculum mapping, logic models) iii. Faculty-oriented (perceptions on student or program success)
iv. Graduate-oriented (perceptions from people who have graduated from the program)
v. Employer-review (perceptions from those employing students following graduation)
e. Do you think this framework could be adopted for use in our One Health programs? G) Evaluation Components: (check all that apply)
a. Curriculum map b. Logic model
c. Assessment (e.g., use student grades as an evaluation tool) d. Feedback (e.g., gather opinions from students/faculty) e. Other (text box)
f. Did the authors provide a step-by-step procedure to follow for evaluating learning outcomes? Yes/No (text box, explain)
Critical appraisal of individual sources of evidence: The sources included in this scoping review will not be critically appraised.
Synthesis of results: Tables and figures will be used to provide descriptive analysis of study characteristics, program context and program discipline, and the type of evaluation methodology. RESULTS
Characteristics of sources of evidence: Data tables will be used to record study statistics on eligible research articles that will include study characteristics (year of publication, year of study conduct, country of conduct), study design, and program context and discipline. However, scoping review methodologies enable modification of data characterization elements as necessary during the review process.
Results of individual sources of evidence: This scoping review aims to characterize the available literature regarding methodologies used to evaluate learning outcomes in competency-based educational programs and curriculum at the post-secondary level (i.e., diploma, undergraduate, graduate, doctoral, or professional) at a university or college. We will report on evaluation methodology strategies outlined in eligible studies, such that we can formulate an evidence-based framework for evaluating learning outcomes in One Health programs in Canada.
DISCUSSION
Summary of evidence: We will use this scoping review to present a summary of research that reports the use of methods for evaluating learning outcomes in competency-based educational programs at the post-secondary level (i.e., diploma, undergraduate, graduate, doctoral, or professional level) at a university or college. The results obtained from this review will help to identify gaps in the current body of evidence, which will provide direction for future research into methods to assess learning outcomes in competency-based education. We will also identify whether we can adapt or adopt these methods to assess learning outcomes in One Health courses or programs at Canadian universities.
Limitations: We will discuss limitations at the study and review level. REFERENCES
Arksey H and O’Malley L. 2005. Scoping studies: towards a methodological framework. Int J Soc Res. 8(1):19-32. https://doi.org/10.1080/1364557032000119616.
Berns JS. 2010. A survey-based evaluation of self-perceived competency after nephrology fellowship training. Clin J Am Soc Nephrol, 5(3):490-496. https://doi.org/10.2215/CJN.08461109 Berrian AM, Wilkes M, Gilardi K, Smith W, Conrad PA, Crook PZ, Cullor J, Nyatanyi T, Smith MH, Kazwala R, and Mazet JA. 2020. Developing a global One Health workforce: the “Rx One Health Summer Institute” approach. EcoHealth, 17:222-232. https://doi.org/10.1007/s10393-020-01481-0
Britten N, Wallar LE, McEwen SA, and Papadopoulos A. 2014. Using core competencies to build an evaluative framework: outcome assessment of the University of Guelph Master of Public Health program. BMC Med Educ, 14(158):1-6. https://dx.doi.org/10.1186%2F1472-6920-14-158
Endnote X7. 2018. Clarivate Analytics, Philadelphia, USA.
Gouvea JS, Sawtelle V, Geller BD, and Turpen C. 2013. A framework for analyzing interdisciplinary tasks: implications for student learning and curricular design. CBE Life Sci Educ, 12(2):187-205. https://doi.org/10.1187/cbe.12-08-0135
Hamilton K, Middleton JL, Pattamakaew S, Khattiya R, Jainonthee C, Meeyam T, and Hueston W. 2020. Mapping veterinary curricula to enhance World Organisation for Animal Health (OIE) Day 1 competence of veterinary graduates. J Vet Med Educ, 41(s1):74-82. https://doi.org/10.3138/jvme-2019-0109
Hodgson JL, Pelzer JM, and Inzana KD. 2013. Beyond NAVMEC: Competency-based veterinary education and assessment of the professional competencies. J Vet Med Educ, 40(2):102-118. https://doi.org/10.3138/jvme.1012-092r
Joyner HS. 2016. Curriculum mapping: a method to assess and refine undergraduate degree programs. J Food Sci Educ, 15(3):83-100. https://doi.org/10.1111/1541-4329.12086
Lam BH and Tsui KT. 2016. Curriculum mapping as deliberation – examining the alignment of subject learning outcomes and course curricula. Stud High Educ, 41(8):1371-1388. https://doi.org/10.1080/03075079.2014.968539
Larsen R and Reif L. 2019. Leveling the core competencies of public health nursing to evaluate senior baccalaureate nursing students. Public Health Nurs, 36(5):744-751. https://doi.org/10.1111/phn.12636
Leibbrandt L, Brown D, and White J. 2005. National comparative curriculum evaluation of baccalaureate nursing degrees: a framework for the practice based professions. Nurse Educ Today, 25(6):418-429. https://doi.org/10.1016/j.nedt.2005.03.005
Luvira V, Bumrerraj S, and Srisaenpang S. 2018. Formative evaluation and learning achievement in epidemiology for preclinical medical students. Indian J Community Med, 43(4):298-301. https://dx.doi.org/10.4103%2Fijcm.IJCM_191_18
Negandhi H, Negandhi P, Tiwari R, Sharma A, Zodpey S, Kulatilaka H, and Tikyani S. 2015. Developing core competencies for monitoring and evaluation tracks in South Asian MPH programs. BMC Med Educ, 15(126):1-8. https://doi.org/10.1186/s12909-015-0403-5
Nousiainen MT, Mironova P, Hynes M, Takahashi SG, Reznick R, Kraemer W, Alman B, Ferguson P, and The CBC Planning Committee. 2018. Eight-year outcomes of a competency-based residency training program in orthopedic surgery. Med Teach, 40(10):1042-1054. https://doi.org/10.1080/0142159X.2017.1421751
Togami E, Gardy, JL, Hansen GR, Poste GH, Rizzo DM, Wilson ME, and Mazet JA. 2018. Core competencies in One Health education: What are we missing?. NAM Perspectives, Discussion Paper, National Academy of Medicine, Washington DC. https://doi.org/10.31478/201806a Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, Moher D, Peters MD, Horsley T, Weeks L, Hempel S, Akl EA, Chang C, McGowan J, Stewart L, Hartling L, Aldcroft A, Wilson MG, Garritty C, Lewin S, Godfrey CM, Macdonald MT, Langlois EV, Soares-Weiser K, Moriarty J, Clifford T, Tunçalp O, and Straus SE. 2018. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med, 169(7):467-473. https://doi.org/10.7326/M18-0850
