Assessment-Driven
Continuing Professional Development
Implications for Physicians, Assessors, Regulators and Certifying Colleges
Author: Dr. Kevin Imrie Date: April 7, 2014
Conflict of Interest Declaration
v I have been a CPSO peer-assessor
v I was Vice-President, Education for the Royal College of Physicians and Surgeons and am their president-elect
v I have no financial relationships with members of pharmaceutical or medical supply companies.
v I do not hold any research grants funded by industry.
v I do not serve on an advisory board of any for- profit industry.
Learning Objectives
At the end of the presentation, you will:
• Be knowledgeable about the role of CPD in ensuring ongoing competence
• Be able to reflect on the role the
assessor plays in contributing to CPD
• Be familiar with the movement towards competency-based education in training and in practice
Competency / Competencies
• An observable ability of a health professional
• Reflects a spectrum
© 2009 Royal College and The International CBME Collaborators
Competent
• Possessing the required
abilities at a specified stage of medical education
• Is always qualified by a frame of reference
Competency-Based Medical Education
…is an outcomes-based approach to the design, implementation,
assessment and evaluation of a
medical education program using an organizing framework of competencies
© 2009 Royal College and The International CBME Collaborators
IOM report, 2001
Health care harms patients too
frequently and routinely fails to deliver its potential
benefits.
8
Persistent Gap in Performance
Consistent evidence of failure to
translate research findings into clinical practice
• 30-40% patients do not get
treatments of proven effectiveness.
• 20–25% patients get care that is not needed or potentially harmful.
Grol R (2001). Med Care
Persistent Gap in Quality of Care
Adherence to recommended health care indicators provided to adults
Content Areas
Preventative care 54.9%
Acute care 53.5%
Chronic care 56.1%
McGlynn A (2003). NEJM
Persistent Gap in Patient Safety Quality of care concerns in hospitals
• Adverse events occur in
2.5 – 16.6%
of all hospital admissions
• At the Ottawa Hospital adverse event rate was 12.7% with 38% deemed
preventable*
- 61% of the events occurred prior to hospital
Forster et al CMAJ April 13, 2004
Traditional CME
Our data show some evidence that interactive CME sessions that enhance participant activity and provide the opportunity to practice skills can effect change in professional practice and, on occasion, health care outcomes. Based on a small number of well-
conducted trials, didactic sessions do not appear to be effective in changing physician performance.
Davis D, JAMA 1999
Role for Assessment in CPD
Assessment is an educational imperative Informs:
• Individuals, groups/teams with data or
information to identify unperceived needs
Guides
:• Identification and development of learning plans through providing credible feedback
Assessment pivotal to training
Assessment in Practice
Two Key Questions
1. Is there are role for self-assessment within the spectrum of assessment strategies or options?
2. What does the literature say about the efficacy of formal assessment
strategies in promoting learning,
enhancing competence or improving performance
?
Accuracy of Self-Assessment
Results
17 of 725 articles met inclusion criteria 20 comparisons between self and
external measures
• 13 demonstrated little, no, or an inverse relationship
• 7 demonstrated a positive association
Davis D , JAMA. 2006
• While suboptimal in quality, the preponderance of
evidence suggests that physicians have a limited ability to accurately self-assess.
• The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment.
18
Self-Assessment in Practice
Key Messages
‘Personal, unguided reflection’ or any global judgment of one’s ability or
performance in a particular domain is…
1. Poorly performed
2. Conceptually flawed
3. Unlikely to be enhanced through training or education!
(Read:“it’s a waste of time!”)
Reflecting “ in action ”
Reflecting “ on action ”
Self-Assessment in Practice
How accurate are decisions to pause and learn before acting?
“Knowing when to look it up: a new conception of self-assessment ability.”
Eva, Regehr, Academic Medicine 2007
Focus on:
• Situational awareness when at the limits of knowledge, ability, experience OR
• When confidence in ability is lacking.
Self-Monitoring in Practice
Definition of Self-Monitoring
“the moment-by-moment awareness of the likelihood that one has the requisite knowledge / skills to act in a particular situation.”
Conclusion:
Greater accuracy in self-monitoring than any global aggregation of performance.
Eva and Regehr Adv Health Sci Educ Theory Pract. 2011
Self-Monitoring of Physicians
Focus for Future Research
1. Exploring the concept of self-guided learning
.
Brydges: Academic Medicine Oct 2010
2. The quality of questions and use of resources in self-directed learning:
personal learning projects in the
maintenance of certification program.
T Horsley: JCEHP 2009
Assessment for Learning
Wh at is the role for peer or personal assessments of competence or performance within mandatory systems of
continuing professional
development?
Physician Assessment
Royal College Strategic Plan
Included two important goals:
1. Promote competence and performance assessment through engaging in lifelong learning
2. Develop standards for the assessment of performance of physicians in practice.
26
Assessment in CPD:
Some Key Assumptions
Assessment strategies and processes must be :
1. Embedded within the practice context and supported by the health system.
2. Relevant to every dimension of
professional practice and across all content domains.
3. Supported within environments that are safe and divorced from threats of
litigation or fear of failure.
Assessment in CPD:
Some Key Assumptions
Assessment strategies and processes must be :
4. Applicable for individuals, groups or inter-professional health teams.
5. Able to generate data and provide constructive feedback.
6. Able to facilitate the identification of
areas where further learning should be focused.
Categories of Assessment
1. Multi-Source Feedback 2. Simulation
3. Audit and Feedback 4. Learning Portfolios 5. Information Systems
Multi-Source Feedback
Research literature has established … 1. Reliability
2. Feasibility and cost effectiveness 3. Educational impact if feedback was
• Credible, specific and accurate and/or
• When coaching was provided
Sargeant JCEHP 2011 Miller BMJ 2010
PAR: Physician Achievement Review
http://www.par-program.org
Multi-Source Feedback
Conclusions
• Valuable formative assessment strategy for individual physicians.
• Empiric evidence to establish reliability in content areas applicable to all physicians.
• Relevant to observable behaviors in three CanMEDS Roles: Communicator,
Collaborator, Professional.
• Quality of the data / feedback provided influence and guide physician response.
Audit and feedback
Impact on performance and health outcomes is small to moderate
Relative effectiveness is enhanced when:
1. Baseline compliance with recommended practice was low!
2. Feedback is provided:
• By a colleague or supervisor
• More than once
• Delivered in both written/oral formats
• Identifies targets with an action plan
Centrality of Feedback…
Impact of workplace based assessment on doctor’s education and performance:
a systematic review
Miller, BMJ 2012
“Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses.”
34
Audit and Feedback
Conclusions
• Valuable assessment strategy for individual physicians in their Medical Expert roles.
• Validity (face and content) based on
trustworthy data that is patient specific.
• Multiple systematic reviews have
established the conditions that influence significant behavior change.
• Feedback is central and critical.
• Applicable to multiple sources of data.
Our Current Vision
“Informed Self-Assessment”
Definition
“A set of processes through which
individuals use external and internal data to generate an appraisal of their own
ability”.
Mann K, Sargeant J.
Acad Med 2011
Informed Self-Assessment
Conceptual Model
1. Sources of information
2. Interpretation of information 3. Response to information
4. External and internal conditions that influence each of these steps
5. Tensions arising from competing data and external influences
Sargeant J. Acad Med 2010
36
Informed Self-Assessment
Some key lessons for the future
1. Work place or work environment must support and enable greater access to data with feedback
2. Data and feedback will be essential to maintaining and improving performance and contribute to better health
outcomes.
Some Conclusions
Assessment for Learning in CPD
Requires the intentional integration of multiple strategies that:
1. Support self-monitoring and the
competencies of self-directed learning.
2. Provide credible data with trusted feedback to identify the ‘path to improvement.’
Current status…
Our Conclusions and Next Steps…
1. Assessment must become a
MANDATORY COMPONENT of any lifelong learning strategy for practice.
2. We are developing a scoping review to inform a set of recommendations for our Council on
“strategies and options to affirm the
continued competence of Fellows of the Royal College”.
New MOC Framework
Section 1
Group Learning Section 2
Self-Learning Section 3 Assessment Accredited
Group Learning Planned
Learning Knowledge Assessment
Unaccredited
Group Learning Scanning
Activities Performance Assessment
Systems Learning
MOC Program Changes 1 of 2
(to be reflected in MAINPORT in 2014)
Policy Change
Fellows and MOC Program participants with new MOC cycles starting on or after January 1, 2014 will be required to complete a minimum of 25 credits in each section of the MOC Program over a 5-year cycle.
i.e. min 25 credits in Group Learning, min 25 credits in Self-Learning, and min 25 credits in Assessment across a cycle
Rationale:
• Promotes assessment as a key learning strategy
• Less restrictive than maximums
• Removes the ‘ceiling effect’ of recording credits
MOC Program Changes - further information
To learn more, please visit:
http://www.royalcollege.ca/portal/page/portal/rc/
members/moc/moc_program
For support, please contact the Royal College Services Centre:
[email protected] 1-800-461-9598
42
Click to edit Master subtitle style
Competence by Design (CBD)
A Vision for Competency-based
Education across the continuum
CanMEDS 2015:
Refresh of our competency framework
44
• Practical framework to support competence
across the continuum of a physician’s career.
• Robust implementation plan to support Fellows and medical educators with the roll-out of
CanMEDS 2015.
Competence by Design:
Vision for Competency-based education across the continuum
Conclusions
• Health system and physician performance needs to improve
• Education alone is insufficient
• Self-assessment can be effective, but
needs to be evidence-informed, guided and credible
• A competency-based model of medical
education across the continuum is needed (and on its way)
• You as peer-assessors will play a vital role
Many Thanks to
Dr Craig Campbell Director of CPD
Dr Ken Harris
Executive Director, Office of Education
