Kirkpatrick modification (Steinert et al., 2006)

Steinert et al. (2006) undertook a systematic review of TTT for BEME. As part of this review they have offered a further modification of Kirkpatrick’s framework (1967) (Table 8). As this is at variance with the BEME review published by Issenberg et al. (2005), only one year earlier, I revisit Kirkpatrick. They have also have maintained four levels. However, they have ‘reverted’ to Kirkpatrick’s original terminology of reaction, learning, behaviour, and results. One key difference here it that they have set their endpoint, results, at improving

undergraduate or postgraduate learning rather than an improvement in patient care. They also divided learning and results into subsections as shown in Table 8.

Table 8 BEME levels

Level 1

Reaction Participants views on the learning experience, its organisation, presentation, content, teaching methods and quality of instruction

Level 2A

Learning – change in attitudes Changes in attitudes or perceptions among participant groups towards teaching and learning Level

2B

Learning – modification of knowledge or skills

For knowledge, this relates to the acquisition of concepts, procedures and principles: for skills this relates to the acquisition of thinking/problem- solving, psychomotor and social skills Level

3

Behaviour – change in behaviours

Documents the transfer of learning to the

workplace or the willingness of learners to apply new knowledge and skills

Level 4A

Results – change in the

system/organisational practice

Refers to the wider changes the organisation, attributable to the educational programme, Level

4B

Results – change among the participants’ students, residents or colleagues

Refers to improvement in the student or resident learning/performance as a direct result of the educational intervention

Before considering their report, it is important to review their inclusion and exclusion criteria. Their research question was:

‘What are the effects of faculty development interventions in the knowledge, attitudes and skills of teachers in medical education, and on the institutions in which they work?’ (Steinert et al., 2006, p.449)

They included only developments related to improve teaching. Their inclusion of effect on institutions probably explains their development of level 4A. They excluded studies related to teaching specific content and studies that considered development of trainees as teachers. This diverges from my approach because it excludes doctors still in training. My area of

exploration specifically relates to doctors still in training. This difference explains why many of the papers I use are not included in this review.

They found 2,907 abstracts of which 53 meet their criteria for inclusion. Only 10 reached the level of ‘results’ with only three in 4B, which relates to demonstrable changes in quality of teaching (Skeff et al., 1986; Marvel, 1991; Nathan and Smith, 1992). The seven papers

showing changes, in 4A, related to changes in the organisational structure of teaching. None of these papers demonstrated any impact on patient care.

Furthermore they included one study that had only been reported at a conference (Sachdeva and Kelliher, 1994). They rated this as a study that reached the ‘4B’ level. This was a report that I was unable to obtain. Neither could I find evidence that the authors had published this in the peer reviewed literature. Another paper (Rayner et al., 1997), that they scored as having reached ‘4A’, was interesting because when I reviewed this paper (section 2.5.7) I felt it only reached the level of reaction. Their only empirical data related to measuring the participants’ satisfaction at the time of the course. They did report that they had held four further lunchtime meetings and set up a committee to link with the network of doctors teaching undergraduates. However, their was no further evidence of the effectiveness or impact of these changes. The use of such low levels of evidence serve to confirm the difficulty finding data to demonstrate the effectiveness of faculty development.

As has been discussed there have been many papers published on TTT courses and

interventions. However, few of these papers show a direct link to improving the changes in the educational organisation of teaching or improving the teaching of the learners. They also report a variety of teaching methods and a lack of consistency of course contents.

2.5.10 Summary

In an attempt to improve ‘doctors as teachers’, TTT courses have been developed. When considering the function of doctors as teachers it is important to take into account the varying roles that they undertake. It is important to consider the characteristics of poor and excellent teaching. The courses can then target appropriate areas to improve teaching. The learners value good facilitators, role models, and information providers. The undergraduates tend to value their trainer’s role as a teacher more highly than as a clinician. The postgraduates tend

to value the trainer’s role as a clinician more highly than as a teacher. The postgraduates look for a facilitator, an information provider, and a clinician, who will guide them in learning how to treat patients.

The impact of TTT courses has been widely reported in the literature. There are many papers documenting the value of participating from the perspective of the attendees. In the main, these courses get favourable responses. There is some evidence that the courses can lead to improvements in the organisation of teaching and what the learners learn. It is noteworthy that these studies did not engage or address what it is that makes a ‘good’ teacher. Furthermore, there is, as yet, no evidence directly linking TTT courses to improved patient care.

As can be seen from the discussion, developing education programmes to improve doctor teachers is not easy. Clearly, a lot of effort has been expended on developing all sorts of short TTT courses in different countries. One starting point is the educational cycle with its four phases of; needs assessment, selection of teaching methods, delivery, and assessment of outcome. It would appear from the literature prior to 1990 that many of the courses have been developed by starting with teaching methods set firmly in the classroom. It was not until the 1990s that descriptions of curricula and assessment of outcomes start to emerge.

Even into the 2000s, journals are still publishing descriptive papers of short courses. This is disappointing because some ‘educationalists’ are still designing and delivering TTT courses without critical consideration of the desired outcome. It is also interesting that while most of the courses are delivered within the classroom, the evidence suggests that the postgraduate trainees value clinical teaching above that delivered in the classroom.

The next section moves from delivering teaching within the classroom to consideration of integration into the context in which our trainees are learning.