Understanding the Purpose of Evaluation

It could be argued that one of the most fundamental weaknesses of current evaluation models is that they try to serve multiple purposes for multiple stakeholders with competing and potentially irreconcilable goals. Just as ‘no servant can serve two masters’ [Bible, Mathew 6.24] (1989), then surely no single evaluation model can be expected to cover the whole gamut of evaluation purposes.

I suggest that the evaluation process needs to fulfil three discrete purposes spread across at least two organisational communities. The purposes are to; Check, Steer and

to Challenge and the major stakeholders are the training community and operational management.

5.3.1. Evaluation as a means of Checking

The checking role of evaluation is of interest to both the training community and the operational managers. Trainers need to understand the efficiency of their training offerings in terms of delivering the agreed knowledge and skills to the agreed

community. To do this they must answers questions such as, is the design appropriate? Are the materials of the right level and accessible to all? Are the formal parts of the programme delivered in the most effective manner? Could the programme be improved by adjusting the nature, approach or presentation of any of the materials? Operational managers, by contrast are interested in whether or not the learning experience

complements the activity in the workplace. They wish to know if there are appropriate structures in place to support the learners and help them translate new knowledge and skills into improved practice.

5.3.2. Evaluation as a Steering Mechanism

This phase of evaluation should also be of interest to the same two communities, but this research showed only limited interest in this role on the part of the training community. Steering is about getting in flight feedback so that the experience can be modified along the way. This should be of interest to the trainers because on long programmes they have an opportunity to fine tune materials and delivery in the light of performance. Feedback generally comes in the form of quality statistics from reaction surveys but can also come in the form of test scores from internal or external

accreditation examinations. Both sorts of results can, and often are used very effectively to inform and develop programme material.

It is also possible to reach out into the post training community to survey how useful they have found the training experience and what success they are having in applying the ideas, concepts and techniques in the working environment. These are the sort of steering data that operational managers desperately need so that they can more

effectively design and sustain the practice ground for returning trainees. For it is through collective experimentation in the workplace that new knowledge is developed and skills transfer takes place, without these structures and appropriate coaching and feedback the overall level of professional practice is unlikely to improve. In this research sample, there was not a single case of any programme systematically reaching out to obtain this sort of evaluation data. Discussions and focus groups with practising Learning and Development professionals and analysis of published evaluation case studies also confirm this evidence. Indeed in the few cases where there is evidence of post training

sampling in the workplace the results are invariably used to modify the training materials rather than the receiving environment.

5.3.3. Evaluation used to Challenge Assumptions

The final role of evaluation is to challenge, that is to challenge the assumptions that gave rise to the training requirement. In this role the primary, indeed possibly the only stakeholder is the operational manager. It is the operational manager that diagnoses the performance problem; the operational manager should therefore work with training professionals to identify the extent to which a gap in skills, knowledge or application is to blame or whether the issue is a deeper malaise or a failure in the learning cycle itself. Whatever the decision, it will be based upon a developing understanding of how people, processes and systems work together to produce productive outcomes. Evaluation has a role not just in discerning any subsequent change in outcomes as a result of training but also in testing the assumptions that underpin how the outcomes are produced. In this way evaluation can not only prove the value of learning but can also deepen the understanding of core business drivers and their relationship to performance outcomes. In so doing it can only enhance the effectiveness of critical business processes.

5.3.4. Refining the Purpose

In proposing a new model of evaluation it is important to understand where current models serve us well and to pay appropriate homage to those models. We have seen that current models serve the training community and very effectively service the ‘check’ purpose. For this reason the supplementary model that is to be presented is designed to be used for and by the operational management community and to specifically address the purposes of ‘steer’ and ‘challenge’ with regard to the deployment of large scale training programmes.

5.4. Understanding the Process of Evaluation