Techniques for Evaluating Residents and Residency Programs

(1)

Techniques

for

Evaluating

Residents

and

Residency

Programs

Thomas G. Quattlebaum, MD

ABSTRACT. A comprehensive, functioning evaluation system is an important component of a residency pro-gram. It should focus on the residency program as well as on the residents and should provide feedback to the residents, their teachers, and the program director. Such a system allows residents and their faculty advisors to receive timely, ongoing, formative feedback concerning resident progress. Likewise, it can help the faculty

rec-ognize the strengths and weaknesses of the training

pro-gram by providing data that can lead to the curriculum changes needed to improve resident experiences. Addi-tionally, it can alert the residency program director to residents whose performance is significantly below that of their peers, allowing intensive faculty and advisor help for these individuals. The practical aspects of pro-viding feedback and implementing an evaluation system apply no less to community-based educational experi-ences than they do to inpatient and outpatient training areas. Pediatrics 1996;98:1277-1283; community-based health care, medical education, resident evaluation, resi-dency training.

Evaluation of residents is an important but

time-consuming task. Its purpose is to improve and to

facilitate development of knowledge and skills in the

learner.1 Achievement of this goal requires

assess-ment of the resident’s progressive acquisition of

pro-fessional knowledge, skills, and attitudes, as well as

documentation of the extent to which the residency

program promotes-or fails to promote-acquisition

of these competencies.

A comprehensive evaluation system should focus

on both the program and the residents. The system

should provide continuous feedback to resident,

teacher, and program director, thereby allowing

res-idents to evaluate their own competencies, teachers

to evaluate their instruction, and program directors

to evaluate their curricula.2

The distinction made by most authors between

formative and summative evaluation is important.

Formative evaluation occurs on an ongoing basis,

and feedback shapes day-to-day behavior. This

use-ful teaching tool resembles a coaching type of

rela-tionship. Summative evaluation is usually done at

the end of an experience to determine whether a

certain standard has been achieved.3-5 Corley’s

exam-ple of the evaluation of an athiete’s performance

illustrates this difference:

Formative evaluation: What specific details were observed

about [Sheila’s] jumping that could help her to improve her

From the Medical University of South Carolina, Charleston, South Carolina.

Acad-emy of Pediatrics.

performance? Summative evaluation: Did Sheila’s

perfor-mance qualify her for, or disqualify her from, the team?2

Both types of evaluation are required in a

resi-dency program. Program directors are required to

make summative judgments of residents’

qualifica-tions for the board certification process. They must

also make decisions regarding retention or

promo-tion of residents. However, in achieving the ultimate

goal-producing capable physicians-thoughtful

medical educators stress the critical importance of

the formative approach, which is geared to

stimulat-ing improved house officer performance.2’’6

For an evaluation system to be effective, several

conditions must be satisfied. Clearly outlined goals

and objectives are needed, and trainees must be

ob-served by evaluators. For the system to be truly

effective for formative evaluation, a trusting

relation-ship must exist between teacher and learner.

Addi-tionally, both the residents and the faculty must see

the system as genuinely useful in contributing to the

overall growth of a resident. If either group thinks

that evaluation does not serve an important purpose,

the system is doomed to failure.

Both community- and university-based educators

should participate in the evaluation process.

Resi-dents especially value the feedback they receive from

their community preceptors, whom residents

per-ceive as real-world, practicing physicians.

Despite the importance of community preceptors

to the evaluation system, several circumstances tend

to hinder their participation. Some practitioners,

aware of the traditional town-gown division, are

anxious about their legitimacy in an academic role.

Additionally, if residents are with them for a

pro-longed period, practitioners often feel uncomfortable

giving critical feedback because of concern about

potential adverse effect on their relationship. Faculty

development programs can help address both of

these issues.7

EFFECTIVE FEEDBACK

Adherence to several principles will enhance the

effectiveness of feedback for residents. As noted by

Davis,6 the dignity and self-esteem of the resident

must be preserved and enhanced. Direct feedback

enhances learning. The longer remediation is

de-layed, the less its effectiveness in changing behavior.

For positive as well as negative feedback, the

evalu-ator and program director should focus on specific

observable behavior rather than on the trainee as a

person. Only one or two points should be focused on

at one sitting; more will fall on deaf ears or will

(2)

points should be coupled with concern and/or

praise for the person being criticized.

An especially effective technique in presenting

negative criticism has been nicknamed the “feedback

sandwich.” In this approach, the positive aspects of

performance are given first (what was done right).

Then the negative point (what was done wrong) is

presented, followed by suggestions of how to

im-prove (what to do next time).

For further information on the techniques of

feed-back, the reader is referred to the excellent

discus-sions of Davis6 and Ende.5

OUTLINE OF AN EVALUATION SYSTEM

A comprehensive evaluation system should

in-dude assessing the educational experience monthly

while assessing the performance of the resident.

Ad-ditionally, it should include periodic assessments of

continuity and other ambulatory experiences, chart

audits, videotaping or observation of patient

encoun-ters, checklists of procedures performed, periodic

evaluation by other professionals such as nurses, and

yearly written and oral in-training examinations.

Numerous evaluation forms are available.2’8 When

designing evaluation forms, it is important to

re-member the practical need to limit forms to either a

single page or two brief ones. Longer evaluation

forms are frequently ignored by both faculty and

residents.

Instructor Evaluation of Residents

At the end of each rotation, designated preceptors

should be asked to rate each resident. For inpatient

and clinic rotations, the rater could be a chief

resi-dent or attending physician in charge of the rotation.

Additional evaluations can be sent to attending

phy-sicians or supervising residents when residents

themselves request such evaluations. For

office-based experiences, the rater should be the resident’s

preceptor. The evaluation form could request an

as-sessment of a resident’s patient evaluation skills,

knowledge, patient care, and personal

characteris-tics. Allowing for an overall rating of the resident’s

potential is also important. Substantial space should

be provided for comments, including the evaluator’s

overall impression of the resident. These comments

are generally valued by residents and program

di-rectors as more helpful than numerical ratings.

For extended experiences or preceptorships, such

as year-round office-based continuity experiences, an

evaluation should be completed periodically,

per-haps monthly or bimonthly.

Resident Evaluation of Rotations

Residents are very willing to provide evaluations

of their rotations and other experiences, especially if

it is clear that the program director and other faculty

members will thoughtfully consider their comments

and suggestions. Feedback from residents about

their experiences can be divided into two areas: the

content of the rotation and their subjective thoughts

about their experience. Both areas are useful for the

program director. For example, a resident might

have had good exposure to important problems on a

rotation but an unsatisfactory experience because of

scheduling or interpersonal problems. The opposite

is also possible: an easy month with little or no

learning experiences.

For a resident’s subjective evaluation of the

rota-tion, a single page should be sufficient to allow for

rating the rotation’s usefulness. A sample form is

shown in “Appendix 1.” This evaluation should

in-dude the resident’s overall satisfaction. Adequate

space should be allowed for written comments, and

residents should be asked to offer at least one specific

recommendation that would make the rotation a

more profitable educational experience.

Addition-ally, it is good to ask whether residents are willing to

have copies of their evaluations sent to rotation

at-tending physicians. Because feedback is often better

accepted and more useful when it is solicited rather

than imposed,9 it can be helpful to ask residents for

the names of attending physicians or supervising

residents from whom they would like to receive

evaluations.

A second brief page can be used for the resident to

assess the content of the rotation by rating the extent

to which its goals and objectives were fuifilled. The

practical necessity of limiting this objective

question-naire to a single page limits the number of objectives

that can be included. A short list of tracer objectives

can be used, assuming that a rotation that meets

these objectives will probably provide a good

learn-ing experience. Cognitive objectives deal with the

extent to which both learning and patient contact

occur. Psychomotor objectives include skills that

res-idents should acquire during the rotation. The

sam-ple form in “Appendix 2” shows an example of

cognitive and psychomotor objectives for one

rota-tion.

Thoughtful faculty members are almost always

interested in improving the educational experiences

of residents, and feedback from a resident should be

shared with the faculty if the resident does not object.

Occasionally, a resident’s comments will be so

strongly negative that the resident will be unwilling

to have the rotation faculty see them with a name

attached. In such circumstances, the resident’s

wishes must be strictly observed, or that

resident-and others when they hear about the

circumstanc-es-will refuse to give any meaningful feedback

about their experiences. However, quarterly or

yearly summaries of resident comments, without

names, can be sent to the faculty while maintaining

resident anonymity.

Other Techniques

Assessment of Continuity and Outpatient Experiences

To provide optimal education for its trainees, a

residency program should provide each resident

with exposure to a broad mixture of health problems:

self-limited to chronic and minor to complex,

includ-ing different age levels and socioeconomic groups.1#{176}

Assessing the success of a program in reaching this

goal requires the collection of data about patients

seen or assigned to trainees, either through a written

(3)

written data on experiences of residents is not

diffi-cult, but useful analysis of this information can be

facilitated by using computers to catalog the data.11

Chart Audits

What residents do, rather than the knowledge they

seem to possess, should provide the most valid data

for the evaluation of performance. Because a

pa-tient’s medical chart is a record of what a resident

actually did, chart audits would seem to be

espe-cially valuable in providing insights into the care the

resident is providing and the nature of the thought

process used in providing this care. Studies have

shown, however, that chart audits cannot reliably

evaluate the quality of care rendered by residents.2’12

Nonetheless, audits do record strengths, weaknesses,

and progress in the quality of a resident’s record

keeping. Acceptable charts do not ensure good

med-ical care, but poor records do suggest sloppy medical

practice.2 Faculty members have a responsibility to

audit the patient charts of residents systematically

and to insist on good records of patient visits.

The biggest practical difficulty in performing chart

audits is the faculty time required to read charts and

to provide written feedback. A suggested approach

is for both community- and university-based faculty

to review officially several resident charts each

month.

Videotaping or Observing Resident-Patient Encounters

Important information about a resident’s

interper-sonal and counseling skills can be obtained by

di-rectly observing a patient encounter, by watching

videotapes, or by using standardized patients.

Ini-tially, residents often are reluctant to have their

pa-tient encounters observed or taped; however, with

time they recognize the benefit of the experience and

become enthusiastic supporters of the exercise.

A major drawback to this technique is the time

required for the evaluator. It is difficult for a busy

community preceptor to take the time required to

perform this task properly. A suggested alternative is

to have trained, nonphysician faculty observe

resi-dent-patient encounters in the preceptor’s office.

The sample form in “Appendix 3” provides the

observer with a suggested outline of important

in-terview stages for use while observing the

patient-resident session and also for teaching and feedback

with the resident after the encounter.

Nurse and Other Staff Evaluations

Nurses and other staff, as well as and more

ex-tended ancillary staff, such as social workers, in the

hospital setting can provide valuable insight into

how a resident is viewed by his co-workers. Nurses

are usually willing to point out positive and negative

aspects of a resident’s performance, commenting on

interpersonal traits that physicians may see

differ-ently, if at all.

Although this information can be useful,

occa-sional residents are reluctant to accept feedback from

nurses and other nonphysician faculty, especially if

the feedback points out some aspects of their

behav-ior that could be improved. Program directors must

support the usefulness of this feedback, seeing that

no retaliation is taken by house officers. Residents

should be reassured that nurses will be asked to

evaluate their attitudes and not their professional

competence.

Procedure Checklists

A log of clinical experiences can be of value in

documenting a resident’s experience in performing

various procedures. This record can be either manual

or computerized. Some programs maintain a listing

of the numbers of times residents perform various

procedures. A more complex system might also

pro-vide for documentation of proficiency in procedure

performance. A faculty member or senior resident

might be required to observe and to confirm that the

resident is competent to perform specific procedures.

In-training Evaluations

A satisfactory base of knowledge of medicine is

required for a physician to be considered competent;

the written in-training examination provided each

year by the American Board of Pediatrics provides

an accurate picture of current knowledge. It has been

suggested, however, that this measurement alone

provides an insufficient basis for gauging the clinical

skills of residents.13 Certainly, facts and knowledge

are important, but equally essential are diligent and

thorough physical skills (eg, physical examinations

and treatment procedures) and affective abilities (eg,

history and interviewing techniques, interpersonal

relationships, and counseling capabilities). These

im-portant areas should not be overlooked when

at-tempting to evaluate a resident’s performance.

Oral in-training examinations and related

meth-ods, such as objective, structured clinical

examina-tions, can be used to assess all three areas of clinical

competence (cognitive, affective, and psychomotor).

Thus, they are not limited to the cognitive area

mea-sured by written examinations.

Although these techniques provide significant

in-sight into a resident’s true skills in pediatrics, their

administration does require more faculty time and

effort than administration of a prepackaged written

examination. This time and effort can yield

impor-tant benefits if the data gathered are used

construc-tively to bring all trainees to an expected level of

competence. Identification of specific areas in which

additional instruction is needed can offset the

addi-tional faculty time required for the exercise, because

future remediation can be focused on areas that can

be acknowledged as appropriate by both the trainee

and the faculty.

THE IMPORTANCE OF TIMELINESS

For feedback from formative evaluation to be most

useful in the educational process, it should be

re-ceived by residents as soon after a rotation is

corn-pleted as possible.5’6’9’14 Increased participation by

both the residents and their instructors can be

ex-pected if the process is timely. For example, if

resi-dents receive feedback shortly after a rotation has

finished, they are much more likely to take the time

(4)

Personal computer programs can be helpful in

providing the quick and accurate feedback that is

needed,8 but photocopies of forms can also be used

to provide rapid turnaround of evaluation forms.

GRADE INFLATION

Faculty members have a noticeable tendency to

give fairly high numerical scores on monthly

evalu-ations. Most residents are given overall scores that

range from the midpoint on a rating scale and

higher. Experience shows that only the exceptionally

weak resident is given a lower rating.8

Despite this propensity for high scores, faculty

members are usually consistent in the relative

scores they give individual residents. With a scale

from “unacceptable” to “outstanding,” the truly

strong residents usually receive “outstanding”

rat-ings, and the average resident normally has a

mix-ture of “very good” and “outstanding” ratings

with an occasional “satisfactory” score. Weaker

residents tend to receive principally “satisfactory”

ratings combined with a few “very good” and no

“outstanding” scores.

Because of this tendency to give high scores,

corn-puter programs that standardize the scores received

by residents can be used. This technique can increase

accuracy and can simplify the comparison of

resi-dents with their peers.

Comments made by faculty members tend to

re-flect accurately the relative ranking of residents.

These comments are often more helpful to the

resi-dents and the program director than the numeric

scores assigned.

COMPUTERIZING THE EVALUATION PROCESS

Because a comprehensive evaluation process

re-quires the manipulation of many data, computer

programs can assist in the task. It is with the

sum-marization of data that the computer proves to be

extraordinarily valuable.8 For example, at each

quar-ter and at the end of the year, the computer can

produce summaries of the data entered, and the

information can be distributed to appropriate

fac-ulty. In addition, a summary of the resident

evalua-tions of each educational experience can be sent to

the responsible attending physician. Copies of the

summaries of instructor evaluations of the residents

can be made available to each resident’s faculty

ad-visor.

Standardized Scores

To assist the program director and the resident

evaluation committee, the computer also can be used

to compute each resident’s mean score on the

eval-uations of that are received during a particular

pe-riod. Using this technique, each resident’s score can

be standardized, simplifying the comparison

be-tween residents and their peers.8

Because some raters tend to give consistently

higher or lower scores than others scoring the same

performance, standardization among raters can

irn-prove the accuracy of the relative ranking of

resi-dents. The computer accomplishes this task by

corn-paring each score received by a resident with the

mean score given by the rater performing the

evalu-ation.

Residents receiving standardized scores in the

up-per or lower 10% among their peers can be flagged

by the computer. Each printout can be useful in

spotting star residents and those with problems. The

stars should receive an extra pat on the back, and

those with problems should receive special help

through their advisors.

IMPLEMENTATION

How evaluation is accepted in a residency

pro-gram is influenced by the manner in which

con-cepts are translated into practice. Corley2 suggests

two very practical procedural principles in the

implementation of an evaluation system. The first

is the need for early resident participation. There is

no better way to implement evaluation than by

requesting that residents assess the quality of their

educational experience on the hospital ward,

where they receive much of their training, before

any other evaluation activity is initiated. Second,

when starting an evaluation program, start slowly.

Any temptation to implement a comprehensive

evaluation program in one fell swoop should be

resisted. If it is introduced in gradual increments,

the program is more likely to be accepted. As

Corley notes:

In practice, there is much to be gained by these two axioms.

If residents become the first evaluators, they quickly acquire

a sense of proprietorship in the program. If faculty, from the

beginning, receive credible information about an important

aspect of their residency program, they will invariably spot

areas where revision is, indeed, desirable. As a consequence,

formative evaluation, which had been unfamiliar to many,

quickly wins a favorable implementation.2

OTHER PRACTICAL ISSUES

One member of the faculty needs to be in charge of

the program’s overall evaluation system. This

per-son, who should support wholeheartedly the

con-cepts of evaluation, must be willing to spend time

and energy in establishing and maintaining the

eva!-uation process. More time will be needed as the

process is being implemented; much less will be

needed once the program is established.

The day-to-day work of running the ongoing

por-tion of the system can be managed by a staff

secre-tary in only a few hours each week. Because yearly

in-training evaluations tend to be more complicated

than the ongoing evaluations, someone on the

fac-ulty needs to assume responsibility for their

imple-mentation.

Faculty advisors should be chosen carefully. Not

everyone on the faculty will be adept at discussing

evaluations with trainees. Residents can be asked

each year whether they would like to keep their

current advisors. If not, they could be allowed to

(5)

APPENDIX I: RESIDENT’S EVALUATION OF ROTATION

This form elicits your evaluation of educational aspects of your rotation. Your frankness will provide information your faculty needs if

our department is to upgrade the quality of your residency training.

Name: ______________________ Service ________________________ Date

__________

List the five most frequent medical problems you saw on this rotation:

(1) (2) (3) (4) (5)

I. In this rotation, rate the attempts to make you aware of the quality of your performance and/or possible ways of improving your skills

and/or increasing your knowledge.

(1) Almost none (2) Haphazard

____

(3) Good/adequate

____

(4) Excellent

____

2. How many patients did you discuss in some meaningful manner with a resident or attending physician?

(1) Almost none

______

(2) 25%

______

(3) 50%

______

(4) 75%-100% ______

3. How effectively were the patients used for teaching purposes?

(1) Poorly

____

(2) Not very

____

(3) Effectively

____

(4) Very effectively

4. Did you have sufficient opportunity to exercise your own clinical judgment?

(1) None

___

(2) Poor (3) Good or adequate

____

(4) Excellent

____

5-8. Rate teaching proficiency of instructors with whom you had the most contact during the rotation.

Name (first and last) POOR FAIR GOOD EXCELLENT

9. How beneficial was this rotation in preparing you for pediatrics?

(1) No benefit (2) Little benefit (3) Beneficial (4) Very beneficial

____

10. Rate your overall satisfaction with your experience on this rotation.

(1) Very disappointed (2) Disappointed (3) Satisfied (4) Very satisfied

Please offer at least one specific recommendation that could lead to improvement of this rotation. Continue on the reverse side, if

necessary.

Please list those from whom you would like to receive an evaluation.

Attending Physicians Supervising Residents

Are you agreeable to sharing this feedback with the rotation? Yes No

(6)

Cognitive Objectives:

Using the I to 4 ratings in the following keys for AMOUNT LEARNED ABOUT THIS PROBLEM and PATIENT CONTACT, rate each

SPECIFIC PROBLEM.

AMOUNT LEARNED ABOUT THIS PROBLEM PATIENT CONTACT?

I Nothing I None with this problem

2 A little 2 Contact but someone else managed the patient

3 A fair amount 3 Managed this problem without supervision

4 Enough to manage such a problem 4 Managed this problem with supervision

SPECIFIC PROBLEMS AMOUNT LEARNED PATIENT CONTACT

1. Office evaluation of development I 2 3 4 1 2 3 4

2. Language delay I 2 3 4 1 2 3 4

3. Team approach to developmental disabilities I 2 3 4 1 2 3 4

4. Psychological testing 1 2 3 4 1 2 3 4

5. Special education evaluation 1 2 3 4 1 2 3 4

6. Occupational therapy evaluation 1 2 3 4 1 2 3 4

7. Physical therapy evaluation 1 2 3 4 1 2 3 4

8. Family dynamics in childhood disabilities 1 2 3 4 1 2 3 4

0 1-2 3-4 4+

AB C D

APPENDIX II: DEVELOPMENTAL DISABILITIES

Name __________________ Date________

Psychomotor Objectives

Circle the number of times you performed each of the following procedures when unsupervised and when supervised.

SPECIFIC SKILLS UNSUPERVISED SUPERVISED

1. Neurodevelopmental examination 0 1-2 3-4 4+

AB C D

APPENDIX III: OBSERVED PATIENT INTERVIEW FORM

Resident__________________ Monitor__________________ Date___________ Patient

____________________________

1, UNACCEPTABLE; 2, NEEDS TO IMPROVE; 3, SATISFACTORY; 4, VERY GOOD; 5, OUTSTANDING

INTERVIEW STAGE

I. Opening

2. Seated and physically oriented to patient before talking

3. Establishes eye contact with patient

4. Is physically relaxed (no distracting hand, arm, feet, leg motions)

5. Uses appropriate invitation to talk

6. Agenda setting

7. Agenda and purpose mutually agreed on

8. Patient’s agreement obtained (overtly or covertly)

9. Problem clarification

10. Adequately clarifies each problem

I 1. Considers psychosocial and family dimension of each problem

12. Completes clarification of each problem before moving on

13. Problem resolution/feedback

14. Provided feedback to each problem

15. Appropriate management of plans presented for each problem including somatic,

educational, psychosocial, where appropriate

16. Provided clear information and effective patient education when applicable

17. Questioned patient as to understanding and agreement

18. Set clear follow-up agenda

19. Closing

20. Terminates medical-/problem-related discussion

21 . Ends encounter in socially appropriate manner

22. Escorts patient to door

General (observed in any/all stages) 23. Patient or resident’s time used efficiently?

24. Resident/patient roles adequately clarified?

25. Quality of rapport established and maintained?

26. Affect and nonverbal behavior congruent with content of messages?

27. Resident notices and uses patient’s lack of congruence, if any, between affect and

content of message

OTHER COMMENTS: (PLEASE USE OTHER SIDE IF NEEDED)

YES NO RATING

12345

12345 12345 12345 12345 12345

(7)

REFERENCES

1. Langsley DG. Evaluation during residency. In: Lloyd JS, Langsley DC,

eds. How to Evaluate Residents. Chicago, IL: American Board of Medical Specialties; 1986:11-30

2. Corley JB. Evaluating Residency Training. 2nd ed. Lexington, MA:

Col-lamore Press; 1983

3. Voytovich AE, Rippey RM, Mathews DA. Deciding how to evaluate

performance. In: Lloyd JS, Langsley DC, eds. How to Evaluate Residents.

Chicago, IL: American Board of Medical Specialties; 1986:57-74

4. Robinowitz CB. Summary remarks. In: Lloyd JS, Langsley DC, eds. How

to Evaluate Residents. Chicago, IL: American Board of Medical Specialties; 1986:173-176

5. Ende J. The evaluation product: putting It to use. In: Lloyd JS, Langsley DC, eds. How to Evaluate Residents. Chicago, IL: American Board of Medical Specialties; 1986:96-116

6. Davis JS. The evaluation process. In: Lloyd JS, Langsley DC, eds. How to Evaluate Residents. Chicago, IL: American Board of Medical Specialties; 1986:75-98

7. Roberts KB, DeWitt TG. Faculty development of pediatric practitioners:

complexities in teaching clinical precepting. Pediatrics. 1996;97:389-393

8. Quattlebaum TG, Sperry JB. A computerized system for evaluation of

residents and residency experiences. Am

I

Dis Child. 1988;142:758-762 9. Ende J. Feedback in clinical medical education. JAMA. 1983;250:777-781

10. Given CW, Browne M, Sprafka RJ, Breck EC. Evaluating primary

am-bulatory care with a health information system. JFam Pract. 1981;12: 293-302

11. Quattlebaum TG. Microcomputer analysis and management of

resi-dency training experiences. Comput Methods Programs Biomed. 1985;20:

169-172

12. DeRouville WH. Peer review in biliary tract surgery. NY State JMed.

1971;71 :1544-1548

13. Quattlebaum TG, Darden PM, Sperry JB. In-training examinations as

predictors of resident clinical performance. Pediatrics. 1989;84:

165-172

14. Burg FD. Deciding what to evaluate: a program director’s checklist for

(8)

1996;98;1277

Pediatrics

Thomas G. Quattlebaum