Relationships between level 1 (reactions) and level 2 (learning)

The Kirkpatrick approach implies causality between levels in that positive (level 1) reactions lead to greater learning (level 2), which in turn leads to transfer and behavioural changes (level 3), resulting in positive organisational change (level 4). Although the Kirkpatrick approach provides little detail about the causal linkages between the levels, a causal relationship is implied (Bates 2004). For example,

If training is going to be effective, it is important that trainees react favourably.

(Kirkpatrick 1994)

However, several studies have failed to confirm the implied causal relationships between levels. For example, one study found that trainee satisfaction was not related to learning and that learning was unrelated to behavioural change (Noe and Schmitt 1986). This study involved the evaluation of a training programme designed to improve the inter-personal and administrative skills of 60 educators.

Similarly, another study showed that there was no relationship between

enjoyment and learning (Warr and Bunce 1995). In this study, 106 trainees, who were junior managers, undertook a four-month open learning programme, which was a new initiative to provide wide-ranging training for all line managers in an organisation. The training comprised structured group sessions as well as self- directed learning.

Two meta-analytical studies corroborated the findings of other studies (Alliger and Janak 1989; Alliger, Tannenbaum et al. 1997). In their meta-analyses, Alliger et al reported that the overall average correlation between reactions of any type and immediate learning was only 0.07. Affective reactions (enjoyment) alone did not correlate (on average, the correlation was just about zero) with immediate learning. However, there were indications that utility reactions (relevance) correlated somewhat with immediate learning but again, the correlation was weak.

It has been proposed that learning may only take place when the trainees are challenged to the point that the training becomes 'unpleasant'. If this is the case, negative correlations would be expected between reactions and learning.

Indeed, negative correlations between level 1 reaction and level 2 learning measures have been found in some educational classroom research (Alliger and Janak 1989).

The Aliger et al. meta-analyses questioned whether level 1 reaction measures should be considered separately from the other levels. They also suggested that reactions should not be used as a surrogate for the assessment of learning.

In this study, level 2 evaluations were carried out by asking the trainees to record (at the end of the training days) whether they had learned new information and new skills.

In the present study, learning information only correlated significantly with the venue and with learning skills. The correlations between learning information and enjoyment and between learning information and relevance were not

finding does not support the causal relationship between levels, which is implied in the Kirkpatrick approach. Rather, it corroborates the findings of the two meta- analyses outlined above (Alliger and Janak 1989; Alliger, Tannenbaum et al. 1997).

Learning skills correlated most strongly (and significantly) with relevance and with, among other criteria, enjoyment. In addition, the relationships between learning skills and relevance and learning skills and enjoyment were both significant. In contrast to learning information, these findings support the causal relationship implied in the Kirkpatrick approach: Participants who found the training enjoyable and relevant also learned new skills and vice-versa. As learning skills correlated most strongly with relevance, the results also

corroborate the Alliger et al. meta-analyses in that utility (relevance) reactions correlate more strongly with learning than affective (enjoyment) reactions.

There was a weak positive correlation between learning information and

learning skills. This suggests that participants who learned new information did not necessarily learn new skills and vice-versa. This supports Bloom's

Taxonomy of Learning in that there are different types of learning (Bloom 1956). Some participants learned in the cognitive and affective domains only, some in the psychomotor domain only and others learned in all domains.

In the present study, the relationship between learning information and reported ease was not significant. However, there was a significant relationship between learning skills and reported ease. A higher proportion of professionals, who did not learn skills than did learn skills, reported that the training had been very easy. This suggests that professionals who found the training very easy did not necessarily learn new skills. This is corroborated in part by the results of

Pearson's correlations. There were negative, although not significant, correlations between ease and both learning information and skills. This

indicates that professionals who found the training very easy, were less likely to learn new information or new skills.

The results of this study question the causal relationships between levels 1 and 2, as implied in the Kirkpatrick approach, e.g. positive 'reactions' at level 1 lead to increased learning at level 2 (Bates 2004). Although there was a relationship

between enjoyment and learning skills, there were stronger links between the relevance of the training and learning both information and skills. This suggests that the relevance of the training is more reliable than enjoyment to determine whether learning takes place.

It is suggested that it is beneficial to evaluate at level 1 in terms of affective reactions (enjoyment), as initially proposed by Kirkpatrick. However, utility (relevance) reactions need to be the focus of evaluations if they are to be robust and in order to determine whether learning has taken place.