Evaluation phase

Evaluation is the means by which a training program‘s outcomes are compared to the set objectives with the aim of finding out the extent to which the training process has achieved its purpose (Armstrong, 2009). A particular training is tailored to suit the needs of a specific time (Staley, 2008), and evaluation studies the level at which the training program meets the set targets through activities such as organisational performance changes, training content and design evaluation, and changes in learners (Alvarez, Garofano & Salas, 2004).

Although post-training evaluation is crucial, evaluation can be done at any stage of the training process. The aim of evaluation is to guide the training activity to ensure its effectiveness, while correcting any deficiencies identified during implementation. To carry out effective evaluation, it is necessary to identify and develop evaluation criteria to guide the process. The process of identifying and developing evaluation criteria will be discussed in the following section.

3.5.1 Developing evaluation criteria

The effectiveness of the training program should be a major concern for training providers and policy makers, because it guides them when considering the relevance and suitability of the activity. Evaluation and effectiveness of training can be treated as synonymous, but Alvarez et al. (2004) identified three differences between them. First, evaluation is a procedural method for determining learning outcomes and provides insights into the training results, while training effectiveness is a hypothetical methodology for comprehending those results with the focus being on the whole learning system. Second, evaluation is used to identify the training outcomes as an individual‘s improvements in learning and job performance, while effectiveness is indicated by whether an organisation has benefitted from such learning. Third, evaluation outcomes define what took place at the end of the training program, while effectiveness results describe why those outcomes were achieved in order to guide experts to develop interventions that will improve the training.

65

The accurate definition of goals and objectives of the training program determine the evaluation procedures and constructs that form the basis of a comprehensive assessment. This varies depending on different target groups‘ emphasis, and between different countries. For instance, a major VET goal of different governments is to empower young people to adapt to a dynamic environment through life-long training; for the workers, however, the emphasis could be wage increases, to the employer, improved productivity, to the trainee, employment and wages, while from a policy-maker‘s view it is to decrease societal expenses (Fretwell, 2003). Fretwell (2003) therefore proposed a framework that identifies how broad evaluation occurs in various constituents of VET schemes. The framework includes:

 defining the objectives

 measuring external VET outputs

 use of transnational standards and valuations

 assessment procedures and concerns

 corroborating the outcomes of VET evaluation.

Developing evaluation criteria assists in identifying the part of the T&D model to be evaluated by defining the expectations that the program is expected to achieve, the means of obtaining the data required for evaluation and the data analysing methods (Armstrong, 2009). Over the years, different researchers have developed models of training evaluations, four of which are discussed in the following section.

3.5.2 Models of training evaluations

The following section examines four different models for measuring the success of T&D programs. First, in 1959, Kirkpatrick published a four-dimensional evaluation method that tests reactions, learning, behaviour and results—a simple measurement method for comprehending training evaluation, and the most quoted procedure (Kirkpatrick & Kirkpatrick, 2005). In this model,

. . . learning is measured during training and refers to attitudinal, cognitive, and behavioral learning. Behavior refers to ―on–the–job‖ performance and, thus, is measured after training. Additionally, reactions to training are related to learning, learning is related to behavior, and behavior is related to results. (Alvarez et al., 2004, p. 388)

66

The Kirkpatrick model has stood the test of time and is widely used to evaluate T&D programs in different fields. For example, van Eerde et al. (2008) used the four levels to measure the effect of training programs to organisational effectiveness, while Piyali et al. (2011) used only the reaction level to do the same. The four dimensions measure different aspects of the training program because they progress from the stage of simply getting a reaction to the training to the more complex processes of measuring results. This model is discussed in more detail in the following chapter.

The second model was developed by Tannenbaum, Mathieu and Martineau, (1993) who added post-training attributes to Kirkpatrick's model and separated evaluation outcomes into transfer and training performances. This model was tested by Alvarez et al. (2004) when constructing ―An integrated model of training evaluation and effectiveness‖. The authors used evaluation measures such as training and transfer performance, and cognitive learning; and effectiveness variables such as pre-training experience and self-efficacy, and post training interventions, such as mastery orientation and learning principles.

In the third model, Holton (1996) incorporated three evaluation objects: transfer, learning and results. He did not consider reactions as a main outcome of training; he regarded them as an intervening or regulating outcome between trainees' learning drive and the actual learning. Thus, learning is linked to transfer, which in turn is associated with the outcomes.

Finally, in the fourth model, Kraiger (2002) provided a model that emphasises three objectives for an evaluation program: learning material and design (i.e. delivery, strategy, and rationality of training), learners‘ behavioural modifications (i.e. emotional, intellectual, and developmental) and structural benefits (i.e. transferability, work performance, and outcomes). The contribution of the fourth model is that it advances the measurement of the learners‘ behaviour changes as a result of the training. These measurements can be quite challenging because it is difficult to tell if the learners‘ emotion or intellect change is as a result of training.

A different perspective in training evaluation was advanced by Kearns and Miller (1997) who advocated for a ‗return on investment‘ (ROI) as a means of assessing the overall impact of training in organisational performance, arguing that particular measures should be used to evaluate specific training, for example customer satisfaction or return on sales, which should improve by at least per cent (Alvarez et al., 2004). Armstrong (2009) adds that besides concentrating on the traditional levels of evaluation there is a need to ―concentrate more on the validation of the total learning process and on the outcomes of learning, which means focusing on

67

the return on expectation‖ (p. 696). This could be achieved through attributes such as increased customer satisfaction, volume of sales and increased production. But this kind of measurement is difficult; in some occupations performance characteristics are difficult to identify and even when identified they cannot always be attributed to training (van Eerde et al., 2008). Attributes such as sales volumes may be dictated by outside factors like security, climatic conditions or the rate of inflation.

The models of evaluation discussed above assist in conceptualising how, where and why evaluation measures can be used. The Kirkpatrick model that was first published in 1959 and composed of four levels reactions, learning, behaviour and results is simple and most comprehensive for use in a T&D model, because the other models components can be integrated into its four levels. For instance: post-training attitudes proposed by Tannenbaum (1993) and transfer of learning proposed by Holton (1996) can be integrated in the behaviour phase; learning material and design, learners‘ behavioural modifications and structural benefits learning materials proposed by Kraiger and Kearn (1997)..

3.5.3 Transfer performance

Transfer performances are measurable behavioural changes of job performance that result from training and can be judged through performance appraisal systems or job behaviour assessments (Alvarez et al., 2004). The transfer performance involves applying knowledge to situations or circumstances that are different from one that is not construed (e.g. learning to operate a piece of equipment or to apply a procedure similar to procedures one has been trained in) (Kitainge, 2009). Studies show that, despite the superior quality of a specific training program, the amount of competency transfer from the training venue to the workplace can either be very low or nothing at all (Donovan & Darcy, 2011; Wagonhurst, 2002). Thus, if a training program aims at facilitating transfer of skills and the application to the work environment, it is vital for trainers and supervisors to perform a thorough determination of the individuals' goals and performance objectives (Chiaburu & Tekleab, 2005); otherwise, workers may not regard the learning they obtain as training for skills if it neither teaches transferable skills nor contributes to the individual‘s educational growth (Armstrong, 2009).

Improving and maintaining workers‘ productivity requires continuous updating of their competencies to enable them adapt and take advantage of opportunities as they emerge. In addition, there is need to ensure that the skills developed are actually used at the workplace (Buchanan, 2006), because employers do not see the need to raise salaries for employees who have trained in skills that do not raise productivity (Cooney, 2002). However, sociopolitical

68

dynamics in a country affect employee participation because their employability depends not only on skills quality, but also on the economic and social dynamism and the environment in which they can be supplied and applied to the workplace (Kingombe, 2008). Thus, there is need to appreciate that the formation of skills is an aspect of the development of democracy, and the interrelationship that exist between the social capacity for learning and innovation, within the state, civil society and at the regional level (Tikly, 2003).

3.6 Summary

In this chapter a discussion of the phases of a generic organisational T&D model has been presented and discussed in the context of application to the MVRSI in Kenya. The model has been divided into three parts derived from several sources of literature: T&DNA, training activities and training evaluation. In the T&DNA phase, the needs of different organisations are analysed to determine whether a training program is necessary to address the organisational goals. Training objectives are then formulated to guide the training program. In the activities phase, decisions touching on the training content, activities, training venues, program designs, duration of training and the presenters are made; and then the training program is implemented.

In the evaluation phase a discussion was presented based on three areas: developing evaluation criteria, models of training evaluation and transfer performance. Evaluation is necessary because it indicates the level at which the training program has been able to meet its objectives, through measures such as post-training organisational performance, program, design evaluation, and attitudinal changes in learners

In the following chapter the three phases of an organisational T&D model are expanded to discuss the identified areas that this research will study. These areas have been identified based on the combined literature from Chapter 2 that analysed the vocational situation in Kenya and the T&D literature in this chapter.

69

CHAPTER 4 DEVELOPMENT OF AN INDUSTRY TRAINING AND DEVELOPMENT