The relationship between competence and performance: implications for assessing practice performance

The relationship between competence and performance:

implications for assessing practice performance

J-J Rethans,

J J Norcini,

M Baro´n-Maldonado,

D Blackmore,

B C Jolly,

T LaDuca,

S Lew,

G G Page

& L H Southgate

Objective This paper aims to describe current views of the relationship between competence and performance and to delineate some of the implications of the dis-tinctions between the two areas for the purpose of assessing doctors in practice.

Methods During a 2-day closed session, the authors, using their wide experiences in this domain, defined the problem and the context, discussed the content and set up a new model. This was developed further by e-mail correspondence over a 6-month period.

ResultsCompetency-based assessments were defined as measures of what doctors do in testing situations, while performance-based assessments were defined as meas-ures of what doctors do in practice. The distinction between competency-based and performance-based methods leads to a three-stage model for assessing doctors in practice. The first component of the model proposed is a screening test that would identify doctors at risk. Practitioners who ‘pass’ the screen would move on to a continuous quality improvement process aimed

at raising the general level of performance. Practitioners deemed to be at risk would undergo a more detailed assessment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice.

Conclusion We propose a new model, designated the Cambridge Model, which extends and refines Miller’s pyramid. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence, the influences of the individual (e.g. health, relationships), and the influences of the system (e.g. facilities, practice time). The model pro-vides a basis for understanding and designing assess-ments of practice performance.

Keywords clinical competence⁄ *standards; physicians, family⁄ *standards; education, medical, continuing⁄ *standards; quality of health care⁄ standards.

Medical Education 2002;36:901–909

Introduction

Most medical students start their careers as qualified doctors after successfully completing the final high stakes examinations. Traditionally, doctors have been

regarded as competent enough to start working with patients immediately. Furthermore, it has been assumed that they would remain competent throughout their professional careers by taking postgraduate cour-ses and that this, together with working in actual practice, would provide sufficient opportunities and evidence to ensure that they remained fit to practise.

However, there are or at least two reasons why this view does not reflect the reality for medical practition-ers today.

Firstly, several studies have shown that there are differences between what doctors can do in controlled high stakes situations and what they really do in actual practice.1–5 Moreover, the relationship between dem-onstrating competency in examinations and behaviour in actual practice appears at the least to be problematic. Studies have shown both high and moderate as well as very low ()0Æ04) correlations between results of doctors performing examinations during tests and in actual practice.1–5 One of the problems with these

1

Skillslab, Maastricht University, The Netherlands,2Psychometrics

and Research, American Board of Internal Medicine, Philadelphia,

Pennsylvania, USA, 3Faculty of Medicine, University of Alcala´,

Madrid, Spain, 4_{Evaluation Bureau, Medical Council of Canada,}

Ottawa, Canada,5Centre for Medical and Health Sciences Education,

Monash University, Victoria, Australia,6_{National Board of Medical}

Examiners, Philadelphia, Pennsylvania, USA, 7Royal Australian

College of General Practitioners, Melbourne, Australia,8_{Faculty of}

Medicine, Division of Educational Support and Development, College of Health Disciplines, University of British Columbia,

Vancouver, Canada, 9_{Centre for Health Informatics and}

Multi-Professional Education, Medical School, University College

London, UK

Correspondence: Jan-Joost Rethans, MD PhD, Skillslab, Maastricht University, PO Box 616, 6200 MD Maastricht, The Netherlands. Tel.: 00 31Æ43Æ388Æ1790; Fax: 00 31Æ43Æ388 4127; E-mail: j.rethans@ sk.unimaas.nl

conflicting results is that authors do not make use of the same implicit or explicit definitions of both test condi-tions and methods used. The situacondi-tions under which candidates are investigated and the descriptions of these situations are not uniform: competence (viour), competency, test behaviour, professional beha-viour (actual) performance and test performance are only some of them. In addition, some studies refer to direct observations of doctors while others use opinions of peers and compare these. Yet these studies claim to describe the same results. We urgently need to establish a common terminology before we can make it possible to determine the implications which these results have for quality assurance or licensure programmes for medical professionals.

Secondly, it is now known that merely undertaking postgraduate courses throughout a professional career, even if done from personal initiative, is not enough to remain working as a ‘competent’ doctor.6This is not only because of the rapid changes in medical information technology. In addition, the growth of government and patient influences (satisfaction rates, legal rights, etc.) reflected in the quality improvement literature shows that additional factors direct the behaviour of a medical professional. This is reflected in the content of different national and international medical standards or guide-lines programmes. The requirements of patients, society and peers make a professional career, and its assessment, much more complex now than it has been in the past.

In this paper we propose a model for the assessment of performance and competence in actual practice with the following characteristics. It will:

• contain clear definitions;

• take into account current knowledge about assess-ment;

• be an international model, more or less independent of any national setting, and

• promote the assessment of an individual medical professional.

The approach to the model, how to use it for screening, the issue of poor performers, the implica-tions of this model and its strengths and weaknesses form the basis for the rest of the paper. Implementation is not part of this paper.

Definitions

In this paper we will use the following definitions: • competency-based assessment measures what

doc-tors can do in controlled representations of profes-sional practice;

• performance-based assessment measures what doc-tors do in actual professional practice.

Although at first glance these definitions may seem wide open, their implications are not. Many research papers on assessment refer to ‘performance-based assessment’ and then describe candidates taking objec-tive structured clinical examinations (OSCEs) and clinical physical examinations (CPXs). We propose that all assessments under examination-like settings should now be referred to as competence-based assess-ments, whereas assessments in actual practice should be referred to as performance-based assessments.

Each setting has its own dynamics. Current know-ledge about the test practice in both settings is shown in Table 1.

In general, competence assessment is easier to administer. However, the perspectives of patients and society demand that doctors should meet the assess-ment standards in their working conditions in any given situation. In future, the emphasis should lie on the assessment of performance.

A model for assessing practising doctors

Figure 1 depicts a model for assessing practising doctors.

The model is founded upon an up-to-date under-standing of what is known about assessing both clinical competence and clinical performance. The model is intended to be applicable across medical specialties and across international boundaries. It is sensitive to the need to be fair to the profession and its members, yet Key learning points

Competency-based assessment measures what doctors can do in controlled representations of professional practice.

Performance-based assessment measures what doctors do in their professional practice.

Assessing doctors in practice should best be done using a three-stage model.

The staged model consists of:

• a screening phase for all doctors, focusing on real practice;

• a continuous quality improvement phase for those who pass the screen, and

responsive to the public’s need to be assured that doctors’ performances are effective. We propose to apply it in a 5-year cycle, preferably dovetailing it with current national standards and using contemporary knowledge on psychometrics.

The model for assessing practising doctors has three components. These are:

1 a general screening component in which all practi-tioners will participate, followed by either

Figure 1 Outline of the authors’ approach to performance assessment.

Table 1Current knowledge of practices for competence and performance assessment

Competence Performance

Available technology Technology evolving

Reasonable costs High costs

Traditional methods Traditional methods rejected

Measurement qualities (theory) known Subjectivity versus objectivity

‘Experimental’ context Naturalistic context

Logistically feasible Logistically difficult

Quantitative Descriptive⁄qualitative

Professionally-based Involves patients

2 a continuous quality improvement component for practitioners who perform well on the screening com-ponent, or

3 a rigorous diagnostic investigation and follow-up component for practitioners who perform poorly on the general screen.

The context of the model

Most existing arrangements for the monitoring of individual practitioners involve either or both of two contrasting approaches.7 In each case, formal mecha-nisms are created for inspection of the practice performance of doctors. But there are numerous and significant differences. For example, the approach favoured in the US, arising from its tradition of entrepreneurial medicine, focuses on identification of the (allegedly few) poor performers and seeks to apply sanctions in the form of restrictions to the license to practice, including revocation.8 Elsewhere, notably in Canada, equally formal mechanisms are imposed, but the intention is described as continuous promotion of quality among all practitioners.9 In the UK, mainly because of recent health care alarms and public concern over the quality of practising doctors, both systems will be in operation from 2003.10

Whatever the context, there are indications of increas-ing support for implementation of systems of monitorincreas-ing doctors’ practice performance.11Such systems are now operational.12Accordingly, this present climate affords an opportunity to consider the most desirable features of any system of practice performance assessment.

The screening component

The arguments made here reflect several guiding principles. In a complex social and professional

environment, efficient use of resources is imperative. This must translate into a mechanism that meets the real challenge of balancing cost with effective measure-ment of performance. Hence the adoption of a screen-ing philosophy that rigorously meets these constraints. An appropriate approach is to consider the task as a screening plus further investigation strategy. If relat-ively quick and efficient screening procedures can be developed, then only a limited number of potentially ‘at risk’ doctors need be subject to further, more detailed scrutiny. One way of achieving this might be by screening and scrutinising routinely collected data on items such as those outlined in Table 2 (e.g. patient reviews for communication skills, health checks for fitness to practise, prescribing behaviour and referral letters for appropriate management).13 These screens could be spread over 3–5 years so that a snapshot of each area is taken, thereby reducing the need for large scale screens every 5 years. Some of this might even be managed on a self-assessment or local peer assessment basis, with appropriate quality assurance devices (sampling) included. Data on the predictive validity of these screens could then be collected. Of course, it may be that quick and effective screening for behaviour as complex as practice performance just does not exist. Hence, screening should assess broadly, but with priority on practice performance. In epidemiological terms, the sensitivity of the first phase must be high. It would be more acceptable to have doctors wrongly classified as being at risk in the first phase, and then reclassified, after the next assessment stages, as good performing colleagues, than to have them classified as good performers who, in the end, are found to be deficient. Finally, the screening assessment must be consistent with, and possess sufficient rigour to cor-rectly classify doctors for, components 2 and 3 des-cribed above.

Table 2Potential screening methods and elements of performance assessment

Methods Elements

Peer review Technical skills; team relationships professional behaviour; ethics;

cognitive skills; patient management

Patient review Communications; professional behaviour; impairment; satisfaction

Performance appraisal Technical competence; resource use; status of privileges

Professional development⁄continuing

medical education (CME)

Active participation in CME; professional behaviour

Compulsory health checks Physical fitness; cognitive function; psychological impairment

Chart audits or competence assessments Use of computerized records; decision-making; prescribing patterns; insight

Aggregate measures (e.g. patient immunisation rates) Standing in context of practice standards

Prescribing behaviour Knowledge currency; clinical decisions; patient management; impairment

Assumptions of screening assessment

Before we describe the specific components of an effective screening assessment, we should also define four complementary resources that we believe must be made available. These include:

1 a doctors’ health programme;

2 definitions of scope or standards of practice;

3 guidelines and standards for the profession, and

4 baseline information.

Potential screening methods

While the premise of any screening is proactive in orientation, the process must be acceptable to the medical community and credible to the public. We acknowledge the importance of conventional indica-tions of possible practice problems, such as patient complaints and actions following hospital review. Reactive measures

In most countries, patient complaints constitute a very large volume of information and they characterise much of the communication between ordinary citizens and licensing bodies. A small fraction of these complaints are deemed worthy of intensive investigation. Although they have risen in number in recent years, it remains true that only a very small number of patient complaints (including malpractice suits) result in adverse judge-ments or disciplinary action.14Performance reviews by hospital oversight committees may result in modifica-tion of privileges, or (less often) their terminamodifica-tion. In US jurisdictions such actions are reportable to the licensing authority, which may undertake independent investiga-tion as well. However, the credibility of a complaints system must remain low because it is neither truly peer-referenced nor self-regulating.

Proactive measures

A portfolio of proactive methods should represent the centrepiece of any proposed performance screening assessment. This is because of the professional value that imposes the obligation on doctors to be know-ledgeable and proficient throughout their careers.

There is a large array of possible screening methods and they vary in feasibility, utility and cost. In the interests of completeness we have included all of them here, irrespective of their potential for contributing meaningfully to the screening process.

Care should be taken, however, that ‘statements of competence’ are applicable to real practice. The major elements of the initial screening process should become a routine part of a doctor’s professional life, consisting

of regular monitoring of performance (e.g. via patient satisfaction reviews, referral rate comparison groups) and more formal periodic assessments (e.g. via chart reviews, peer reviews). While the screening component is intended to focus on the assessment of performance, it could also include some low cost, easily administered competency assessments (e.g. written examinations). Similarly, while Table 2 includes important, possibly generic elements of professional performance, other elements of performance should not be excluded. For example, the important domain of clinical judgement may not be assessed adequately by those methods listed.

Given the greater challenge of obtaining reliable performance assessments, competency assessments would add ‘reliability’ to the assessment process. However, this screening phase should ultimately emphasize performance assessments. To ensure rigour, competency assessments should be administered under controlled conditions. The screening component assumes that programmes to assess doctors’ health are already in place. If they are not, they should be encompassed in the screening process prior to periodic performance or competency assessments so as to rule out health-related causes of poor performance.

Continuous quality improvement

The second component of the model focuses on learning, as its thrust is one of continuous quality improvement (CQI) for practitioners who perform well on the screening component. It is expected that over 90% of all practitioners would be directed to this component of the model from the screening compo-nent. This CQI component would build upon the results of the assessment strategies comprising the screening component of the model, and would be largely self-guided by the practitioner. This adds another argument in support of the proposal that the first screening phase should primarily be based in and on real practice: doctors will view the results of their performance data as ‘of their own’. This sense of personal identification with the subject can make an important contribution towards stimulating change. In this second component, practitioners could employ a reflective learning model to construct their learning around their screening assessment results and their individual practices. Any assessment activity built into this component need only be of minimal rigour, its purpose being to guide the practitioners to areas of strength and weakness in their own performances.

Although CQI depends largely on the doctor’s self-guidance, it will be apparent if doctors remain in this

component or transfer to other components because the initial screening phase is to be repeated periodically.

Rigorous performance assessment

The purpose of the third component of the model is to obtain a detailed assessment of a practitioner who performed poorly on the screening as a basis for decisions on remediation, rehabilitation or exit from the profession. It is expected that less than 10% of practitioners will be directed to this component. Given the high stakes involved for both the practitioner and society, the assessment process in this component must be of the highest rigour. The standards applied in this component should preferably be at least national ones and be approved by bodies and⁄or boards of the profession(s) involved.

This third component also involves the major task of seeking out doctors who are still remediable. Once again, the focus here is on assessing performance. However, this will be complemented by an equally rigorous assessment of competence, which will serve as a mechanism for understanding poor performance, and for developing remedial strategies for correcting poor performance. Whereas component 2 of the model was largely self-directed by the practitioner, component 3 will be directed by a professional or government body responsible for the quality of practitioners’ perform-ance. We propose a staged assessment also in this phase, to be first performance-based and then competency-based. In order to understand why doctors perform poorly, we need to ask ‘why’ and ‘how come’ questions so that we can identify factors or reasons behind poor performance such as impairment, behavioural and health-related issues. These questions should be based on the outcomes of the first screening phase. The answers should direct the examiners firstly to explana-tions for poor performance (such as current⁄ tempor-ary⁄difficult personal circumstances) and secondly to specific assessment areas where candidates show inad-equate results. At the end of this phase, more traditional competency-based assessment strategies may be used, such as detailed knowledge tests, standardised patient examinations of clinical and practical skills, structured oral tests, tests of cognitive abilities, direct observation of doctor–patient contacts and so on.

This approach has been adopted by the UK General Medical Council (GMC) and has been implemented for the entire profession.10 The introduction of the GMC’s Performance Procedures in July 1997 has enabled the identification of doctors whose perform-ance is seriously deficient, calling their registration into

question. This completes the GMC’s facilities for dealing with dysfunctional doctors, which were previ-ously limited to conduct or health-related matters. The performance of doctors who enter the procedures is assessed in two parts, comprising peer review of practice to assess performance in the workplace and tests of competence to assess knowledge and skills basic to that performance. Equivalent assessments for peer review for all registered doctors have been devised and piloted. The competence assessment always includes a written examination and at least one practical exam-ination, but the written examination may be an objec-tive (multiple-choice) test, a short-answer test, or some form of essay test. The practical examination(s) may involve interactions with standardised patients, clinical models, or other forms of simulations. Trained non-medical assessors recruited by open advertisement in the national press take part in both phases alongside the medical assessors. By spring 2001, 230 doctors had entered the programme. General practitioners are represented in proportion to their numbers in the profession, and other disciplines that have been assessed include surgeons, emergency doctors, anaes-thetists, psychiatrists, paediatricians and radiologists. The Cambridge Model for Performance and Competence

Miller proposed an assessment model with four stages or levels, designated ‘knows’, ‘knows how’, ‘shows how’ and ‘does’, as shown in Fig. 2.15

Miller refers to ‘shows how’ as being an assessment of performance. However, as argued earlier in this paper ‘shows how’ nowadays should be referred to as com-petency-based assessment rather than as performance-based assessment. Miller’s triangle implicitly assumes that competence predicts performance. However, the exact relationship between competence and perform-ance is complicated.1–5,10 Factors such as time pres-sure, day of the week, mood of the patient and doctor and impact of the fore-going examination influence clinical performance just as individual deficient com-petence on specific domain-related knowledge areas does.

Miller’s figure is very useful in educational settings, especially for setting up medical curricula or courses for students. The four stages can easily be used to build an educational programme that begins with the assimil-ation of pure knowledge (reading books and articles) and progresses through the acquiral of clinical skills to development of real performance in practice. However, the model is no longer very helpful in terms of assessment in real practice.

Another drawback of Miller’s figure is that it fails to account for the influences of other factors on clinical performance. While these influences are numerous, they may be arbitrarily classified as either system-related or individual-system-related.

System-related influences include government pro-grammes and initiatives, patient expectations, guide-lines or policies developed by the practice facility, time, and accessibility to other health professionals.

Individual-related influences include the physical and mental health of the doctor, their state of mind at the time of the performance assessment, and their relationships with others, including patients, peers (including other health professionals) and their own family.

Miller’s triangle is a static figure, whereas contem-porary assessment demands a more flexible figure. It is vitally important that our staged assessment pro-gramme allows us to view the performance of an individual doctor from different angles. These different angles will allow, for example, fuller answering of the ‘why’ questions submitted to the assessee.

To illustrate this and the interaction between these systemic and individual influences and competence and performance, we propose a modification to Miller’s triangle (Fig. 3).

This effectively inverts Miller’s model. Clearly, competence is an important prerequisite for perform-ance and is therefore represented in the main triangle in the centre. Alternatively, this may be conceptualised as the beam of light which competence sheds on perform-ance, but which does not of itself illuminate the whole picture. We propose at least two further triangles, or shafts of light, to illuminate the influences of the system-related and individual-related factors that should also considered when assessing the performance of a doctor. In fact, it is very likely that more shafts of light are necessary to wholly illumine the performance of an individual doctor. The corollary is that not all the problems related to doctor performance will be explained by competence alone. Because of this, it may be more cost-effective in the first instance to examine other systemic or individual influences before conducting rigorous assessment of doctor competence. Figure 2 Miller’s Triangle15_.

Figure 3 The Cambridge Model for delineating performance and competence.

Conclusion

The purpose of this paper was to describe current views of the relationship between competence and perform-ance and to delineate some of the implications of this relationship for assessing doctors in practice. Compe-tency-based assessments were defined as measures of what doctors do in testing situations, while perform-ance-based assessments were defined as measures of what doctors do in practice. We argued that tests of competence are feasible and efficient, and produce results with known, valid properties. In contrast, methods of assessing performance are just evolving but hold greater promise of meeting the needs of patients and society by describing the actual quality of care delivered by doctors.

The distinction between competency-based and performance-based methods leads naturally to a model for assessing doctors in practice. The first component of the model we propose is a screening test, a relatively quick and efficient procedure that would identify doctors at risk. Practitioners who ‘pass’ the screen would move on to a continuous quality improvement process aimed at raising general levels of performance. Practitioners at risk of poor performance would undergo a more detailed assess-ment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice.

Finally, the competence⁄performance distinction traces at least part of its heritage to Miller’s model, which divided methods according to whether the examinee is required to know, know how, show how or do. We propose the Cambridge Model, which extends and refines Miller’s work. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence combined with the influences of factors related to the individual (e.g. health, relationships) and factors related to the system (e.g. facilities, practice time). The model provides a basis for understanding and designing assessments of practice performance.

Contributors

All authors contributed equally to the discussions leading to this paper, which were undertaken during the 10th Cambridge Conference. J-JR and JJN undertook main responsibility for preparing the draft, co-ordinat-ing input from the other authors and writco-ordinat-ing the final version of the paper.

Acknowledgements

Grateful acknowledgement is made to the sponsors of the 10th Cambridge Conference, the Medical Council of Canada, the Smith & Nephew Foundation, the American Board of Internal Medicine, the National Board of Medical Examiners and the Royal College of Physicians.

Funding

There was no external funding for this study.

References

1 Rethans JJ, Sturmans F, Drop R, van der Vleuten C, Hobus P. Does competence of general practitioners predict their per-formance? Comparison between examination setting and

actual practice.BMJ1991;303:1377–80.

2 Ramsey PG, Wenrich M, Carline JD, Inui TS, Larson EB, Logerto JP. Use of peer ratings to evaluate physician

per-formance.JAMA1993;269:1655–60.

3 Ram P, van der Vleuten C, Rethans JJ, Schouten B, Hobma S, Grol R. Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination

for actual medical performance in daily practice.Med Educ

1999;33:197–203.

4 Southgate L, Campbell M, Cox J, Foulkes J, Jolly B,

McCrorie Pet al.The General Medical Council’s

perform-ance procedures: the development and implementation of

tests of competence with examples from general practice.Med

Educ2001;35:20–8.

5 Page GG, Fielding DW. Performance on PMPs and

per-formance in practice: are they related?J Med Educ

1980;55:529–37.

6 Davis DA, Thompson MA, Oxman AD, Haynes RB. Chan-ging physician performance. A systematic review of the effects

of continuing medical education strategies.JAMA

1995;274:700–5.

7 Dauphinee D, Fabb W, Jolly B, Lansley D, Wealthall S, Procopis P. Determining the content of certifying

examina-tions. In. Newble DI, Jolly BC, Wakeford RE, eds.

Certifica-tion and RecertificaCertifica-tion in Medicine: Issues in the Assessment of Clinical Competence.Cambridge: Cambridge University Press; 1994.

8 Norcini JJ. Recertification in the United States.BMJ

1999;319:1183–5.

9 Dauphinee WD. Revalidation of doctors in Canada.BMJ

1999;319:1188–90.

10 Southgate LH, Pringle M. Revalidation in the United King-dom: general principles based on experience in general

prac-tice.BMJ1999;319:1180–3.

11 Southgate LH, Cox J, David T, Hatch D, Howes A, Johnson Net al.The assessment of poorly performing doctors: the development of the assessment programmes for the General

Medical Council’s Performance Procedures.Med Educ

2001;35(Suppl.):2–8.

12 Cunningham JPW, Hanna E, Turnbull J, Kaigas T, Norman GR. Defensible assessment of the competency of the

practis-ing physician.Acad Med1997;72:9–12.

13 Davies HA, Crossley J, Howe A, Jolly BC, Newble DI. Per-formance assessment using outpatient letters: the Sheffield

assessment instrument for letters.Med Educ2001;35:1115–

24.

14 Page GG, Bates J, Dyer SMet al.Physician-assessment and

physician enhancement programmes in Canada.J Can Med

Assoc1995;153:1723–8.

15 Miller GE. The assessment of clinical skills ⁄

compet-ence⁄performance.Acad Med1990;65(Suppl.):S63–S67.

Received20March2002; editorial comments to authors13June 2002; accepted for publication17June2002