Competency-based continuing professional
development
CRAIG CAMPBELL1, IVAN SILVER2, JONATHAN SHERBINO3, OLLE TEN CATE4& ERIC S. HOLMBOE5, FOR THE INTERNATIONAL CBME COLLABORATORS
1
Royal College of Physicians and Surgeons of Canada and University of Ottawa, Canada,2University of Toronto,3McMaster University, Hamilton, and Royal College of Physicians and Surgeons of Canada,4Center for Research and Development of Education, University Medical Center, Utrecht, the Netherlands,5American Board of Internal Medicine, USA
Abstract
Competence is traditionally viewed as the attainment of a static set of attributes rather than a dynamic process in which physicians continuously use their practice experiences to ‘‘progress in competence’’ toward the attainment of expertise. A competency-based continuing professional development (CPD) model is premised on a set of learning competencies that include the ability to (a) use practice information to identify learning priorities and to develop and monitor CPD plans; (b) access information sources for innovations in development and new evidence that may potentially be integrated into practice; (c) establish a personal knowledge management system to store and retrieve evidence and to select and manage learning projects; (d) construct questions, search for evidence, and record and track conclusions for practice; and (e) use tools and processes to measure competence and performance and develop action plans to enhance practice. Competency-based CPD emphasizes self-directed learning processes and promotes the role of assessment as a professional expectation and obligation.
Various approaches to defining general competencies for practice require the creation of specific performance metrics to be meaningful and relevant to the lifelong learning strategies of physicians. This paper describes the assumptions, advantages, and challenges of establishing a CPD system focused on competencies that improve physician performance and the quality and safety of patient care. Implications for competency-based CPD are discussed from an individual and organizational perspective, and a model to bridge the transition from residency to practice is explored.
Introduction
In an era of increasing demands for greater physician accountability, improved patient safety, and better quality of care, the introduction of strategies to enhance physician competence and produce measurable outcomes is reasonable and desirable. The goal of continuing professional develop-ment (CPD) is to ensure that physicians possess the required knowledge, skills, attitudes, and abilities to maintain and enhance competence and improve performance within their professional roles. However, studies of the effectiveness of formal CPD in enhancing competence and performance suggest that a substantial gap persists between the evidence available to inform practice and its translation into improved quality of care (Cabana et al. 1999; McGlynn et al. 2003; Choudhry et al. 2005).
Physicians are motivated to engage in CPD activities by a desire to deliver high-quality care to their patients (Ryan & Deci 2000) and by their commitment to lifelong learning as members of a self-regulated profession. However, as partici-pation in CPD is increasingly mandated by regulatory authorities as a requirement of licensure, there is a growing need to ensure that physicians’ competence and performance are maintained in accordance with professional standards (Federation of Medical Regulatory Authorities of Canada 2008).
In this theme issue, competence is defined as ‘‘the array of abilities across multiple domains or aspects of physician performance in a certain context’’ (Frank et al. 2010). This definition implies that competence is both conditional on and constrained by each physician’s practice context, is dynamic, and continually changes over time. The prevailing premise in
Practice points
Competency-based CPD is:
. a dynamic process that enables physicians to ‘‘progress in competence’’ toward the attainment of expertise within a defined practice context
. enabled by a set of learning competencies that physicians use to reflect on and learn from their practice experiences
. founded on clear, effective and measurable competen-cies for practice and on the integration of assessment and educational strategies that guide the planning of lifelong learning activities
. helpful in addressing the limitations of current approaches to CPD by virtue of its orientation around competencies that enhance performance, improve quality of care, and increase patient safety
Correspondence: Dr. Craig Campbell, Director, Professional Affairs, Royal College of Physicians and Surgeons of Canada, 774 Echo Drive, Ottawa, ON, Canada K1S 5N8. Tel: (613) 260-4173; fax: (613) 730-2410; email: [email protected]
medical education has been that the competence required to practise independently is established upon the successful completion of a postgraduate education program. Success is judged on the basis of the achievement of established learning objectives, specialty requirements, and/or the completion of a certification examination. By this model, competence is a static set of attributes defined by the profession that serves as the basis for licensure. Residents are thus ‘‘launched’’ into practice with the assumption that the capabilities they demonstrate on entry into practice will remain at or above acceptable levels of performance until they retire (Klass 2007).
A competency-based model of CPD does not view competence as a static state. It begins with the premise that residents enter practice with a set of learning competencies that enable them not only to practise independently but also to sustain and expand their capabilities and to acquire new knowledge and skills over time. Applied to the problems presented within each physician’s defined ‘‘scope of practice,’’ these learning competencies serve as the basis for ‘‘pro-gressing in competence’’ toward expertise (Dreyfus & Dreyfus 1986).
This paper explores the intricate challenges and implica-tions of developing a competency-based CPD model by addressing three questions:
(1) What learning competencies must be acquired by all residents before they enter practice?
(2) How will the competencies for practice be defined? (3) What are the advantages of pursuing
competency-based CPD?
What learning competencies must
be acquired before entry into
practice?
Some authors have suggested that, to be effective lifelong learners, graduating residents must enter practice with a defined set of learning competencies (Ellaway et al. 2007; Parboosingh et al. 2008) that will enable them to critically assess and revise their practice through a process of reflection that allows them ‘‘make sense of complex situations’’ and ‘‘learn from experience’’ (Mann et al. 2007). These learning competencies can be described under the five domains summarized in Table 1 and explored in detail in the following sections.
Knowing one’s practice
The first learning competency is the ability to create and use a profile of the problems and issues typical of one’s own practice, thus facilitating the development of a practice-specific learning strategy (Galbraith et al. 2008). This competence enhances the ability of physicians to engage in learning activities relevant to their day-to-day work. For most physi-cians, the number and type of clinical problems they assess, diagnose, and manage will dominate their lifelong learning strategies. However, learning can be linked to other dimen-sions of professional life, such as teaching, research, and administration, addressing areas such as medical ethics, risk
management, patient safety, office management, and health advocacy.
Creating an accurate practice profile entails reviewing and integrating information from a variety of sources, such as electronic health records, patient registries, and claims data. Practice profiles provide a basis for the selection of appro-priate learning activities (e.g., to address gaps in knowledge, create measurable learning objectives, and achieve relevant outcomes measures. Because physicians’ practices evolve through the course of their careers, practice profiles will need to be continuously reviewed and updated.
Scanning the environment
The second learning competency is the ability to systematically and effectively scan one’s environment for new and relevant ideas. Effective scanning enables the physician to identify innovations at the development stage, new evidence that has been reviewed and approved by the profession (e.g., practice guidelines), and old practices that should be discontinued because they have been found to be ineffective or potentially harmful.
As members of a knowledge-intensive profession, physi-cians are expected to absorb and respond to an enormous volume of information scattered among thousands of journals, textbooks, monographs, reports, and guidelines (McKibbon et al. 2004; Haynes et al. 2006; Coppus et al. 2007). A systematic approach to scanning enables physicians to ‘‘filter information based on relevance and validity,’’ thus ensuring that they are reviewing the most clinically useful information (Davidoff & Florance 2000). Physicians can take advantage of electronic knowledge dissemination services that ‘‘push’’ summaries of relevant or critically appraised literature, drug alerts, or practice guidelines. More informal scanning
Table 1. Competency domains for lifelong learning*.
Domain Key competencies
Knowing one’s practice Ability to use practice information to identify learning priorities develop and monitor CPD plans Scanning the environment Ability to access information sources for
innovations in development new evidence for potential
integra-tion into practice
Managing learning in practice Ability to establish a personal knowl-edge management system to store and retrieve evidence select and manage learning projects Raising and answering
questions
Ability to
construct a question search for evidence
record and track conclusions for practice
Assessing and enhancing practice
Ability to
use tools and processes to measure competence and performance develop action plans to enhance
practice
*Adapted with permission of the Royal College from: Parboosingh J, Campbell C, Silver I, Horsley T, Richardson D, Dath D, Brien S, Ergina P, Bankey R, Frank JR; the Scholar Lifelong Learning Working Group. 2008. Pursuing Excellence in Practice: A CanMEDS Scholar Program on Lifelong Learning. Ottawa: The Royal College of Physicians and Surgeons of Canada.
techniques might include reminder systems within electronic health records, attending rounds or conferences, or sharing ideas with peers, colleagues or other health professionals who have similar or shared practices.
Managing learning in practice
The third learning competency is the ability to establish electronic or other personal knowledge management systems as a foundation for determining what information sources to access, what to read in depth, what to scan, and why. For example, RSS feeds (information streams from the Internet tailored to the recipient’s interests or needs) can be set up to deliver the latest information relevant to one’s practice profile, and online bibliographic databases can categorize and store records for easy retrieval. A learning portfolio can be created to select and manage projects to support continuous practice improvement, and online discussion forums can support the exchange of new information and ideas for enhancing practice. Although these strategies imply a significant amount of work, especially in view of the rapid expansion of medical knowledge, physicians must have the competence to adapt to change and continuously improve their performance in lifelong learning (Shaughnessy & Slawson 1999). Physicians require training to set up these personalized learning ‘‘portals’’ (Plsek & Greenhalgh 2001; Ho et al. 2004).
Raising and answering questions
The fourth learning competency is the ability to formulate good questions and translate them into learning opportunities. Asking questions is a frequent and natural activity of physicians (Barrie & Ward 1997). Some questions reflect an immediate need to resolve uncertainty or solve a problem. In the care of an individual patient, for example, questions frequently arise that stimulate an immediate search for evidence on which to base a clinical decision. Other questions are not time-sensitive and are generated to gain greater conceptual understanding or to satisfy intellectual curiosity. Questions can be stimulated by a spectrum of activities and can identify learning needs across multiple competencies (e.g., attending rounds, reading the literature, teaching, or reviewing performance data).
Since physicians typically learn best when learning is contextual, addresses their defined needs, and is directly relevant to their work (Covell et al. 1985; Embell 1999), the ability to pose and answer a well-formulated question is a central learning competency. Unfortunately, many questions raised in practice go unanswered because of a lack of resources at the point of care, insufficient time to search for evidence, and the difficulty of formulating answerable ques-tions (Covell et al. 1985; Barrie & Ward 1997). The use of question formulation tools such as PICO (Population, Intervention, Comparator, and Outcome) may assist in defining search strategies and analyzing evidence in clinical areas (Armstrong 1999; Huang et al. 2006). Strategies to record, track, and resolve (make sense of) questions can facilitate the appraisal and translation of new knowledge into practice.
Practice assessment and enhancement
The fifth learning competency is the ability to use processes and tools to continuously assess and measure the impact of learning on knowledge, skills, and performance in practice. Assessment provides important opportunities for practitioners to assess how they are doing and how current their knowledge is, whether as individuals or as a team. Potential strategies for knowledge assessment include multiple-choice or short-answer questions, simulation (using standardized patients, high-fidelity mannequins, and computer-based simulation), direct observation, and audit and feedback. Processes that provide meaningful data and feedback to identify gaps (with or without comparative data from other physicians with similar practices) can bring unperceived needs to light. CPD plans derived from such assessment strategies can contribute to the achievement of measurable outcomes.
Although physicians’ use of personal practice outcomes data to improve competence and performance remains infrequent (Audet et al. 2008), their ability to independently detect strengths and weaknesses in areas of knowledge, skills, attitudes, or performance as these apply to clinical practice is often inaccurate (Davis et al. 2006). Comparisons of physi-cians’ self-rated assessments with external observations demonstrate little, no, or an inverse relationship (Eva & Regehr 2008). Physicians with the lowest accuracy in self-assessment have been shown to be the least skilled and most confident (Kruger & Dunning 1999). These studies provide evidence of the need to develop new initiatives and formats to assist the self-assessment process and to more accurately assess broader domains of competence such as lifelong learning (Davis et al. 2006).
To address gaps in knowledge, skills, or performance, physicians must have the opportunity and ability to engage in or plan assessments of their own practice, to measure their performance against specific standards of care, and to translate the findings into actionable plans. Unfortunately, no assess-ment option is uniformly effective, and the mean improveassess-ment in performance with existing approaches has been found to be modest – typically, from 5% to 10% (Jamtvedt et al. 2006).
How will the competencies for
practice be defined?
In postgraduate medical education, the specific competencies to be learned and assessed are defined by the training program in accordance with external standards applicable to each discipline. When residents make the transition to independent practice, their training serves as the basis on which they assume responsibility for their actions and decisions as independent and autonomous physicians. They must rely, more than during their formal training, on self-directed learning processes through which they assume greater control over the issues or problems they pursue and how they will evaluate and learn from the outcomes (Candy 1991). In addition, the content they encounter will continually shift as a result of changing referral patterns, emerging research evidence, and personal interests. Given all of these factors,
the establishment and maintenance of professional compe-tencies is clearly a complex endeavour.
Currently, medical regulatory authorities use a variety of strategies to determine whether practising physicians meet minimum standards of performance within a given context. This process is invariably focused on detecting physicians who have become incompetent, rather than on promoting the progression from competence toward excellence. From a regulatory perspective, if physicians are considered to be self-sufficient professionals, and their competence is viewed as a personal set of attributes or abilities, then someone must be held accountable when errors occur (Klass 2007). However, a regulatory approach to defining competence undermines the role of assessment as a professional expectation by generating apprehension about the consequences of the discovery of any area of incompetence.
Equally, the public’s view of competence is not based exclusively on a set of credentials or attributes. The public are primarily concerned with whether physicians possess the requisite knowledge and skills to solve patients’ problems, have a history of good outcomes, and are able to practise effectively within the health system (Klass 2007).
Within the last decade, several alternative approaches to defining the competencies required for practice have been developed and applied. In the United Kingdom, the General Medical Council’s Good Medical Practice document defines a set of general and discipline-specific competences expected by physicians and serves as a model or framework for the development of curricula and assessment strategies across all phases of medical education (General Medical Council 2010). In Canada, the CanMEDS framework describes a set of general competencies that serve as the basis for the development of educational and assessment strategies for residents (Bandiera et al. 2006). These competencies are being promoted as the basis on which CPD activities can be developed and evaluated for practice. However, the domains of competence set out in the CanMEDS framework have been criticized on the grounds that they are too broad and lack clear benchmarks (Del Bigio 2007). In the United States, the Outcome Project steered by the Accreditation Council for Graduate Medical Education in collaboration with the American Board of Medical Specialties has made it possible for assessment in medical schools and graduate training programs to be explicitly competency based (Green et al. 2009). Increasingly, these general competency
frameworks are being applied within
maintenance-of-certification programs in the United States (Hammond et al. 2005; Miller 2005; Green et al. 2009) and within revalidation and recertification strategies in Canada and the United Kingdom (Federation of Medical Regulatory Authorities of Canada 2008; Academy of Medical Royal Colleges n.d.).
Defining clear, effective, and measurable competencies for practice will require the integration of educational and assessment strategies and tools relevant to specific practice contexts. Although general statements of competencies are helpful in defining the focus for continuing medical education, the profession – working in collaboration with regulatory authorities, educational institutes, and the public – will need to be diligent in defining meaningful metrics for competency-based CPD that ensure credibility, transparency, and
accountability. The biggest challenge may be the development of a practical process for recording and monitoring practice competencies that is easy to use, effective as a guide to planning CPD activities, trusted and accepted by physicians and the public, and not subject to punitive review or oversight outside the profession.
What are the advantages of
pursuing competency-based
CPD over the current model?
Accreditation standards for CPD organizations or programs are based in large part on a set of educational criteria that begin with the assessment of some type of need. For most programs, these ‘‘needs’’ are defined by planning committees who reflect the intended target audience. This needs assessment informs the development of learning objectives and the selection of educational methods to address the needs identified. The assumption is that participation in the program will lead to learning that will, in turn, be translated into practice. The impact of formal CPD based on this program planning mode has been studied extensively. Several recent systematic reviews of the impact or effectiveness of group learning on defined outcomes have identified a moderately positive impact of group learning on knowledge but a small to negligible impact on clinical behaviours and patient outcomes (Mansouri & Lockyer 2007; Marinopoulos et al. 2007). Given the limitations of our current system of CPD, there are advantages to re-establishing a CPD system around the competencies that enhance performance, improve quality of care, and increase patient safety.
Competency-based CPD from an individual perspective
Competency-based CPD recognizes the importance of learning that is designed to achieve quantifiable improvements in practice. When residents enter practice equipped with compe-tencies in self-directed learning, they are in a position to develop a CPD plan specific to their practice needs, to continuously improve their performance, and to measure the impact of their learning on their practice and the health system in which they work (Batalden & Davidoff 2007). Competence promotes engagement in learning that is based, in part, on authentic assessments of current knowledge, skills, and abilities. Competency-based CPD is not confined to lecture halls or small-group workshops. It can occur in many settings, such as through the person-to-person exchange of stories and tacit knowledge, in simulation centres, and in practice under the supervision of a mentor or coach. Ideally, learning opportu-nities should be tightly connected with practice, so that they enable individuals to rapidly apply and evaluate in a real-world setting what they have learned through courses or workshops. Equally, assessment in competency-based CPD is not restricted to knowledge, skills, or performance in clinical practice; rather, assessments can apply to multiple competency domains, such as communication skills, collaboration with other health professionals, and aspects of professionalism. Rather than simply documenting participation in learning activities ‘‘for
credit,’’ a competency-based approach to CPD would require physicians to develop learning activities to review or enhance specific competencies to achieve measurable outcomes.
Competency-based CPD from an organizational perspective
Competency-based CPD changes the approach to needs identification from one based on perception and opinion to one derived from performance metrics, health outcomes, and adverse events (among others). The planning process for a competency-based learning event starts with the ‘‘ends in mind’’ and works backward, rather than starting with what physicians perceive they want to learn about and hope that what is learned will be translated forward into practice (Moore et al. 2009). Competency-based CPD integrates the development of learning activities across a broad range of venues, ranging from conferences and rounds to assessments in a real, simulated, or virtual practice. Every learning activity must have several layers of complexity to simultaneously address the needs of different physicians at different stages of progress toward expertise. Competency-based CPD must place greater empha-sis on individual learners, challenging the dominant role of traditional formats such as large-group conferences. Finally, competency-based CPD promotes the measurement of out-comes beyond the satisfaction of learners or the self-reporting of learning; rather, it attempts to measure higher levels of outcomes for practice (Moore 2003). CPD organizations are important contributors to the development of a framework of competencies that extends beyond postgraduate education and facilitates the ability of individual physicians to meet their learning goals.
An alternative approach to competency-based CPD A competency-based approach to postgraduate medical education (PGME) has the potential to reduce the time required to acquire the knowledge, skills, and abilities necessary for entry into independent practice. One such example is the implementation of competency-based postgraduate education in the Orthopedic Surgery program at the University of Toronto, which seeks to disconnect competence attainment from a set time frame (Kramer et al. 2009). Although CBME does not necessarily lead to shorter training periods and earlier entry into practice, this project anticipates that some individuals will achieve competence sooner than others.
With this example in mind, if training time in PGME could be reduced by, for example, two years without compromising the attainment of a defined set of professional competencies necessary for independent practice, the training time ‘‘remaining’’ could be used in a different way to bridge the traditional boundaries between postgraduate education and CPD. Assuming that the competencies established for earlier entry into practice are likely to be at a more ‘‘generalist level,’’ a competency-based approach to CPD could use the time saved during PGME to facilitate the acquisition of more specialized knowledge and skill as practice needs and interests shift over time. For example, residents could choose to devote one day a week over a defined time frame to ‘‘return to school’’ for periods
of intense study. Devoting time to teaching others would both enable the transmission of acquired knowledge and skills to the next generation of learners and model the learning processes by which residents can learn and improve their practice. Devoting time to teaching, along with refining, extending, and attaining new competences beyond those acquired at initial certification, is a natural approach to CPD for professionals (ten Cate & Durning 2007). This commitment to teaching and learning may promote the continuous refinement and attainment of compe-tencies within and even beyond the traditional boundaries of a specialty.
Conclusion
A competency-based approach to education helps to define basic learning competencies that resident physicians need to acquire before they enter independent practice, so that they have the skills to maintain and enhance their competence throughout their professional lives. The competencies of the life-long learner must also be made explicit and at all levels of training from entry into medical school through residency and into clinical practice. Although the task is challenging on many fronts, the development of a competency-based model for CPD, given its important implications for individual learners, CPD provider organizations, and the health care system as a whole, may provide the foundation for developing a new vision of CPD.
Acknowledgement
The authors would like to thank and to acknowledge the work of the contributors to the development of the CanMEDS Scholar Program on Lifelong Learning who developed the modules within the Pursuing Excellence in Practice curricu-lum, including Susan Brien, Tanya Horsley, John Parboosingh, Denyse L. Richardson, Deepak Dath, Patrick Ergina, and Jason R. Frank. The authors express their thanks to Dr. Horsley for her review of this manuscript.
Declaration of interest: The authors report no conflict of interest. The authors alone are responsible for the content and writing of this article.
Notes on contributors
CRAIG CAMPBELL, MD, FRCPC, is a specialist in Internal Medicine, Associate Professor of Medicine at the University of Ottawa and is the Director, Professional Affairs at the Royal College of Physicians and Surgeons of Canada. Dr. Campbell directs the Centre for Learning in Practice; a CPD research and development unit at the Royal College focusing on physician lifelong learning.
IVAN SILVER MD, MEd, FRCPC, is Professor of Psychiatry and Vice –Dean of Continuing Education and Professional Development in the Faculty of Medicine, University of Toronto.
JONATHAN SHERBINO, MD, MEd, FRCPC, is the Director of Continuing Professional Education, Division of Emergency Medicine, McMaster University, Hamilton, Ont., and a Clinician Educator with the Royal College of Physicians and Surgeons of Canada.
OLLE TEN CATE, PhD, is Professor of Medical Education and Director of the Center for Research and Development of Education at University Medical Center, Utrecht, the Netherlands.
ERIC HOLMBOE, MD, is Senior Vice President and Chief Medical Officer at the American Board of Internal Medicine and American Board of Internal Medicine Foundation. He is also Professor Adjunct of Medicine at Yale University, and Adjunct Professor at the Uniformed Services University of the Health Sciences.
References
Academy of Medical Royal Colleges [UK]. n.d. Revalidation. Available from: http://www.aomrc.org.uk/introduction.html (Accessed 6 June 2010). Armstrong EC. 1999. The well-built clinical question: The key to finding the
best evidence efficiently. WMJ 98(2):25–28.
Audet AM, Raiu R, Jacobs CM, Schick JF, Aviles AD. 2008. Transparency as a pillar of a quality and safety culture: The experience of the New York City Health and Hospitals Corporation. Jt Comm J Qual Patient Saf 34(12):707–712.
Bandiera G, Sherbino J, Frank JR, editors. 2006. The CanMEDS assessment tool handbook: An introductory guide to assessment methods for the CanMEDS competencies. Ottawa: Royal College of Surgeons and Physicians of Canada.
Barrie AR, Ward AM. 1997. Questioning behaviour in general practice: A pragmatic study. BMJ 315(7121):1512–1515.
Batalden PB, Davidoff F. 2007. What is ‘‘quality improvement’’ and how can it transform healthcare? Qual Saf Health Care 16(1):2–3.
Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, Rubin HR. 1999. Why don’t physicians follow clinical practice guidelines? JAMA 282(15):1458–1465.
Candy PC. 1991. Self-direction for lifelong learning: A comprehensive guide to theory and practice. San Francisco: Jossey-Bass.
Choudhry NK, Fletcher RH, Soumerai SB. 2005. Systematic review: The relationship between clinical experience and quality of health care. Ann Intern Med 142(4):260–273.
Coppus SF, Emparanza JI, Hadley J, Kulier R, Weinbrenner S, Arvanitis TN, Burls A, Cabello JB, Decsi T, Horvath AR, et al. 2007. A clinically integrated curriculum in evidence-based medicine for just-in-time learning through on-the job-training: The EU-EBM project. BMC Med Educ 7:46.
Covell DG, Uman GC, Manning PR. 1985. Information needs in office practice: Are they being met? Ann Intern Med 103(4):596–599. Davidoff F, Florance V. 2000. The informationist: A new health profession?
Ann Intern Med 132(12):996–998.
Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. 2006. Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA 296(9):1094–1102. Del Bigio MR. 2007. Please slow down the CanMEDS express [letter]. CMAJ
176(6):812.
Dreyfus HL, Dreyfus SE. 1986. Mind over machine: The power of human intuition and expertise in the era of the computer. Oxford: Blackwell. Ellaway R, Evans P, McKillip J, Cameron H, Morrison J, McKenzie H, Mires G, Pippard M, Simpson J, Cumming A, et al. 2007. Cross-referencing the Scottish Doctors and Tomorrow’s Doctors learning outcome frame-works. Med Teach 29(7):630–635.
Embell M. 1999. Information at the point of care: Answering clinical questions. J Am Board Fam Pract 12(3):225–235.
Eva KW, Regehr G. 2008. ‘‘I’ll never play professional football’’ and other fallacies of self-assessment. J Contin Educ Health Prof 28(1):14–19. Federation of Medical Regulatory Authorities of Canada. 2008. Physician
revalidation. Available from: http://www.fmrac.ca/policy/revalidation_ eng.html (Accessed 6 June 2010).
Frank JR, Snell S, ten Cate O, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: Theory to practice. Med Teach 32(8):638–645. Galbraith RM, Hawkins RE, Holmboe ES. 2008. Making self-assessment
more effective. J Contin Edu Health Prof 28(1):20–24.
General Medical Council [UK]. 2010. Good Medical Practice. Available: http://www.gmc-uk.org/guidance/good_medical_practice/ contents.asp (Accessed 6 June 2010).
Green ML, Aagaard EM, Caverzagie KJ, Chick DA, Holmboe E, Kane G, Smith CD, Lobst W. 2009. Charting the road to competence:
Developmental milestones for internal medicine residency training. J Grad Med Educ 1:5–20.
Hammond ME, Filling CM, Neumann AR, Homburger HA, Education Committee, College of American Pathologists. 2005. Addressing the maintenance of certification challenge: The College of American Pathologists Response. Arch Path Lab Med 129(5):666–675.
Haynes RB, Cotoi C, Holland J, Walters L, Wilczynski N, Jedraszewski D, McKinlay J, Parrish R, McKibbon KA, McMaster Premium Literature Service (PLUS) Project. 2006. Second-order peer review of the medical literature for clinical practitioners. JAMA 295(15):1801–1808. Ho K, Bloch R, Gondocz T, Laprise R, Perrier L, Ryan D, Thivierge R,
Wenghofer E. 2004. Technology-enabled knowledge translation: Frameworks to promote research and practice. J Contin Edu Health Prof 24(2):90–99.
Huang X, Lin J, Demner-Fushman D. 2006. Evaluation of PICO as a knowledge representation for clinical questions AMIA. Annu Symp Proc 2006:359–363.
Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. 2006. Audit and feedback: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev Issue 2. No CD000259. Kramer W, Alman B, Reznick R. 2009. Resident training in 2009: It’s the
quality of time and not the quantity that matters. Canadian Orthopaedic Association Bulletin Issue 85. Available from: http://www.coa-aco.org/ coa-bulletin/issue-85/forum-resident-training-in-2009-its-the-quality-of-time-and-not-the-quantity-that-matters.html (Accessed 6 June 2010).
Kruger J, Dunning D. 1999. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psych 77(6):1121–1134.
Klass D. 2009. A performance-based conception of competence is changing the regulation of physicians’ professional behavior. Acad Med 82(6):529–535.
McGlynn EA, Asch SM, Adams J, Keesey J, DeCristofaro A, Kerr EA. 2003. The quality of health care delivered to adults in the United States. N Engl J Med 348(26):2635–2645.
McKibbon KA, Wilczynski NL, Haynes RB. 2004. What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med 2:33.
Mann K, Gordon J, MacLeod A. 2007. Reflection and reflective practice in health professions education: A systematic review. Adv Health Sci Educ Theory Pract 14(4):595–621.
Mansouri M, Lockyer J. 2007. A meta-analysis of continuing medical education effectiveness. J Contin Edu Health Prof 27(1):6–15. Marinopoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH,
Magaziner JL, Miller RG, Thomas PA, Prokopowicz GP, Qayyum R, et al. 2007. Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep). Jan(149):1–69.
Miller SH. 2005. American Board of Medical Specialties and repositioning for excellence in lifelong learning: Maintenance of certification. J Contin Educ Health Prof 25(3):151–156.
Moore DE Jr.. 2003. A framework for outcomes evaluation in the continuing professional development of physicians. In: Davis D, Barnes BE, Fox R, editors. The continuing professional development of physicians: From research to practice. Chicago: AMA Press. pp 249–274.
Moore DR Jr, Green JS, Gallis HA. 2009. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Cont Educ Health Prof 29(1):1–15.
Parboosingh J, Campbell C, Silver I, Horsley T, Richardson D, Dath D, Brien S, Ergina P, Bankey R, Frank JR; the Scholar Lifelong Learning Working Group. 2008. Pursuing Excellence in Practice: A CanMEDS Scholar Program on Lifelong Learning. Ottawa: The Royal College of Physicians and Surgeons of Canada.
Plsek PE, Greenhalgh T. 2001. Complexity science: The challenge of complexity in health care. BMJ 323(7313):625–628.
Ryan RC, Deci EL. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 55(1):68–78.
Shaughnessy AF, Slawson D. 1999. Are we providing doctors with the training and tools for lifelong learning? West J Med 181(5–6):325–328. ten Cate O, Durning S. 2007. Peer teaching in medical education: Twelve reasons to move from theory to practice. Med Teach 29(6):591–599.