Evaluating effects from
workplace training and
employee development
(TED)
Annual HEAD Conference
June 11th, 2007
PhD Student Anders Dysvik
Department of Leadership and Organizational Management
Norwegian School of Management BI
Definition and current interest
(Tannenbaum, 2002; Kraiger & Ford, 2007)
•
Training and employee development (TED) is defined as a systematic
approach to learning and development to improve individual, team, and
organizational effectiveness (Kraiger & Ford, 2007, p. 281).
•
Workplace settings are becoming increasingly complex
•
Pressures from both public authorities and ”customers” in terms of
improved solutions and products
•
Organizations must be attractive for new employees and simultaneously
prevent existing employees from leaving the organization
•
An increasing trend that TED supplements formal education in workplace
settings at the employer’s expense
•
TED advocates are facing increasing demands in demonstrating return on
investments from such activities
•
59 per cent of employees between ages 22-66 participated in TED during their last 12
months of employment (Nyen & Skule, 2005)
•
Annual TED spendings have been estimated to 17,7 Billion NOK, or 2,8 Billion USD
(Nordhaug, Gooderham & Døving, 2004)
Scope
•
Lay theories suggest a causal relation between investments in TED and
organizational performance at macro level (Sels, 2002)
•
Norwegian companies are at the lower end of the scale in systematically
assessing the return of investment relating to TED (Nordhaug &
Nordhaug, 2004; Lai, 2004).
•
Majority of TED studies limited to needs assessment, design,
implementation, evaluation (Salas & Cannon-Bowers, 2001; Wright &
Boswell, 2002).
•
TED at this level is one of the most widespread phenomena studied within the HR
domain (Whitfield, 2000)
•
Cross-sectional designs insufficient in arguing for causal relationships
(Wright, Gardner, Moynihan & Allen, 2005).
•
Cross-sectional designs establishing positive relationships often lack
control of performance prior to participation.
•
When included, past performance is highly correlated with performance following TED
(Arthur jr., Bennett jr., Edens & Bell, 2003; Morin & Renaud, 2004; Maurer, Weiss &
Barbeite, 2003)
Causal assumptions
•
It has been argued that TED is of little or no value for organizations, in that
more than 70 per cent of workplace learning are informal events in the daily
setting (Pfeffer & Sutton, 2000)
•
Vast parts of acquired knowledge and skills may fade into oblivion (Arthur jr.,
Stanush, Bennett & McNelly, 1998)
•
Contrary, recent meta-analyses suggest a standardized rho of 0.60 between
TED and performance (Arthur et al., 2003)
•
TED should be assessed based on employee perceptions and in relation to
the ”bundle” of HR activities (Tannenbaum, 2002; Lee & Bruvold, 2003;
Wright, 2002)
•
Managerial or HR assumptions of TED are not always in line with employee
perceptions of the same practices (Edgar & Geare, 2005)
•
TED should be assessed in relation to the mechanisms it operates, as a
number of factors seems to influence participation, acquirement, and transfer
of TED content (Kuvaas, 2007; Colquitt, LePine & Noe, 2000)
•
Evidence-based TED as opposed to intentions behind the efforts, ”hypes” or
trends which assumedly justify such investments (Pfeffer & Sutton, 2006)
Evaluating effects from TED
•
Both researchers and practitioners agree that
assessing TED effectiveness is important
•
There are clear inconsistencies regarding how
assessments should be conduced (Alliger,
Tannenbaum, Bennett jr., Traver & Shotland 1997).
•
Established evaluation practices are not always
based on evidence-based criteria (Kraiger, 2002)
•
The most wide-spread practice seems to be the one
which hold the least degree of evidence-based
information regarding training effectiveness (Noe,
2002)
Are TED effective?
•
For the last forty years, the dominant tool has been Kirkpatricks hierarchical
model (1959a; 1959b; 1960a; 1960b)
• Practical guidelines based primarily on Kirkpatrick’s own experiences in a time when few other models were available,
•
Although revised and presented in later writings, the model has not changed
significantly.
•
Given the widespread acceptance of the Kirkpatrick hierarchy, any reasoned
discussion of training evaluation should embark here (Kraiger, 2002. p. 333).
•
Higher levels should not be evaluated if satisfactory results at lower
levels have not been attained
•
Direct relationships between the levels of the model
•
Changes at the higher levels are more valuable than at lower levels
Are TED effective II?
•
Scarce support for assuming that lower levels are of less concern than
higher levels
•
The field of learning psychology has evolved since 1960.
•
Lack of conceptual clarity within and between levels
•
Low or absent direct relations between levels 1 and 2
•
In a recent study of evaluation practices among 367 U.S.
companies, 78% used level 1 measures, 32% level 2 measures, 9 %
level 3 measures, and 7% level 4 measures (Van Buren & Erskine,
2002).
•
The widespread use of reaction measures may be explained in that they are
easy to acquire.
•
When regarded in isolation, such reaction measures fail to explain the alleged
relation between employee reactions and consequent performance at
Are TED effective? III
•
Affective reactions such as ”I liked the course” and consequent
performance .02 (Alliger et al. 1997), .10 reactions and knowledge
acquisition/ .09 reactions and skill acquirement (Colquitt et al., 2000).
Recently replicated in a meta-analysis of 162 TED studies (Arthur jr. et al.,
2003)
• Most reactions are positive, whereas, negative reactions demonstrate a positive, albeit weak relationship with performance (Schmidt & Björk, 1992)
• How should effects be assessed, do you wait for positive reactions to occur at level 1 before proceeding to level 2?
•
A strong negative relationship between levels 2 and 3, which call for a more
integrative perspective (Arthur et al., 2003; Colquitt et al., 2000)
•
Assuming impact at level 4 becomes problematic as content increases in
complexity, in addition to:
• Competing hypotheses?
• Controlling for past performance?
•
In the most systematic efforts as of yet, no conclusions could be drawn
between the different levels and impact at macro level (Alliger et al., 1997)
Moving on:
Decision-based evaluation
(Kraiger, 2002; Twitchell, Holton & Trott, 2001)
•
What is the purpose for evaluation?
•
Present evidence for decision-making
•
Feedback to designers and trainers
•
Internal and external marketing
•
What are the obstacles to evaluation?
•
Not required by the organization
•
Additional costs associated with evaluation
•
Lack of time
•
Lack of training in how to evaluate
•
Training function may have everything to lose and nothing to gain from the
data
Moving ahead:
Decision-based evaluering
(Kraiger, 2002)
Moving ahead
Decision-based evaluation II
(Kraiger, 2002)
•
Training content and design
•
Formative purposes
•
Reaction measures
•
”I liked the course” .02, while ”I found the course content relevant” .26
(Alliger et al., 1997)
•
Novice participants may limit the latter (for instance student’s
evaluation of teaching)
•
Training environment, transfer expectation, logistics and
Moving ahead
Decision-based evaluation III
(Kraiger, 2002; Kraiger, Ford & Salas, 1993)
•
Changes in the learner
•
Cognitive outcomes
•
Affective outcomes
•
Behavioral outcomes
•
Changes not necessarily the same as improved job performance, as this is inflicted by a
number of influences
•
Level of retention over time
•
Organizational payoffs
•
Evidence that trainees are applying learned behaviors or skills back at work
•
These behaviors result in more effective performance
Decisions regarding design
•
Agreement between trainers and TED advocates regarding the
level of evidence before initiation
•
Arguing for effects of TED beg the presence of experimental
designs, often difficult to conduct in workplace settings (Morin &
Renaud, 2004)
•
For those without a scolarly approach, insurmountable or not a problem at all
•
For those with a scolarly approach, a challenging endevaour
•
Often conducted as quazi-experimental designs
•
Worst case scenario: Only immediate post-tests, and still arguing
for effects from TED
•
Examples of alternative designs Internal Reference Strategy
(Haccoun & Saks, 1998)
•
Level of design evidence should be in line with the strenght of
claims made by TED advocates
•
Less formal designs may still serve a purpose, and is far better
than direct linkage of reactions and effectiveness, or failing to
evaluate all together
References
•
Alliger, G. M., Tannenbaum, S. I., Bennett jr., W., Traver, H., & Shotland, A. (1997). A
Meta-analysis of the relations among training criteria. Personnel Psychology, 50,
341-358.
•
Arthur jr., W., Bennett, W., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence
skill decay and retention: A quantitative review and analysis. Human Performance,
11(1), 57-101.
•
Arthur jr., W., Bennett jr., W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of Training
in Organizations: A Meta-Analysis of Design and Evaluation Features. Journal of
Applied Psychology, 88(2), 234-245.
•
Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of
training motivation: A meta-analytic path analysis of 20 years of research. Journal of
Applied Psychology, 85(5), 678-707.
•
Edgar, F., & Geare, A. (2005). HRM practice and employee attitudes: different measures
- different results. Personnel Review, 34(5), 534-549.
•
Fischer, C. D., Schoenfeldt, L. F., & Shaw, J. B. (2006). Human Resources Management
(6th ed.). London: Houghton Mifflin Company.
•
Haccoun, R. R., & Saks, A. M. (1998). Training in the 21st century: Some lessons from
the last one. Canadian Psychology-Psychologie Canadienne, 39(1-2), 33-51.
•
Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of ASTD,
13(11), 3-9.
•
Kirkpatrick, D. L. (1959). Techniques for evaluating training programs: Part 2 - Learning.
References II
•
Kirkpatrick, D. L. (1960). Techniques for evaluating training programs: Part 3
-Behavior. Journal of ASTD, 14(1), 13-18.
•
Kirkpatrick, D. L. (1960). Techniques for evaluating training programs: Part 4 - Results.
Journal of ASTD, 14(2), 28-32.
•
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of Cognitive, Skill-Based, and
Affective Theories of Learning Outcomes to New Methods of Training Evaluation.
Journal of Applied Psychology, 78(2), 311-328.
•
Kraiger, K. (2002). Decision-Based Evaluation. In K. Kraiger (Ed.), Creating,
implementing, and maintaining effective training and development: State-of-the-art
lessons for practice (pp. 331-376). San Francisco, CA: Jossey-Bass.
•
Kraiger, K. (2003). Perspectives on Training and Development. In W. C. Borman, D. R.
Ilgen & R. J. Klimoski (Eds.), Handbook of Psychology (Vol. 12 - Industrial and
Organizational Psychology, pp. 171-192). Hoboken, NJ: Wiley&Sons.
•
Kraiger, K., & Ford, J. K. (2007). The expanding Role of Workplace Training: Themes
and Trends Influencing Training Research and Practice. In L. L. Koppes (Ed.), Historical
Perspectives in Industrial and Organizational Psychology (pp. 281-309). Mahwah, NJ.:
Lawrence Erlbaum Associates.
•
Kuvaas, B. (In press). An exploration of how the employee-organization relationship
affects the linkage between perception of developmental human resource practices and
employee outcomes. Journal of Management Studies.
•
Lai, L. (2004). Kompetanse som begrep og fenomen. In L. Lai (Ed.), Strategisk
References III
• Lee, C. H., & Bruvold, N. T. (2003). Creating value for employees: investment in employee development. International Journal of Human Resource Management, 14(6), 981-1000.
• Maurer, T. J., Weiss, E. M., & Barbeite, F. G. (2003). A model of involvement in work-related learning and development activity: The effects of individual, situational, motivational, and age variables. Journal of Applied Psychology, 88(4), 707-724.
• Morin, L., & Renaud, S. (2004). Participation in corporate university training: Its effect on individual job performance. Canadian Journal of Administrative Sciences-Revue
Canadienne Des Sciences De L Administration, 21(4), 295-306.
• Noe, R. A., & Colquitt, J. A. (2002). Planning for training impact: Principles of training
effectiveness. In K. Kraiger (Ed.), Creating, implementing, and maintaining effective training
and development: State-of-the-art lessons for practice (pp. 53-79). San Francisco, CA:
Jossey-Bass.
• Nordhaug, O., Gooderham, P. N., & Døving, E. (2004). Kompetanseutviklingens jernlov. In O. Nordhaug (Ed.), Strategisk kompetanseutvikling-teori og praksis (pp. 322-342). Oslo:
Universitetsforlaget.
• Nordhaug, O., & Nordhaug, I. W. (2004). Kompetansebehov i næringslivet. In O. Nordhaug (Ed.), Strategisk kompetanseledelse-teori og praksis (pp. 302-321). Oslo:
Universitetsforlaget.
• Nyen, T., & Skule, S. (2005). Livslang læring i norsk arbeidsliv. In M. Raabe, O. Raaum, P. O. Aamodt, N. M. Stølen & A. M. Rustad Holseter (Eds.), Utdanning 2005 - deltakelse og
kompetanse. Oslo: Statistisk Sentralbyrå.
• Pfeffer, J., & Sutton, R. I. (2000). The Knowing-doing gap: How smart companies turn
References IV
• Pfeffer, J., & Sutton, R. I. (2006). Hard Facts, Dangerous Half-Truths, and Absolute Nonsense.
Profiting from Evidence-Based Management. Boston, MA: Harvard Business School Press.
• Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress.
Annual Review of Psychology, 52, 471-499.
• Schmidt, R. A., & Bjork, R. A. (1992). New Conceptualizations of Practice - Common Principles in 3 Paradigms Suggest New Concepts for Training. Psychological Science, 3(4), 207-217.
• Sels, L. (2002). 'More is not necessarily better': the relationship between the quantity and quality of training efforts. International Journal of Human Resource Management, 13(8), 1279-1298.
• Tannenbaum, S. I. (2002). A Strategic View of Organizational Training and Learning. In K. Kraiger (Ed.), Creating, implementing, and maintaining effective training and development:
State-of-the-art lessons for practice. (pp. 10-52). San Francisco, CA: Jossey-Bass.
• Twitchell, S., Holton, E. F., & Trott, J. R., Jr. (2001). Technical training evaluation practices in the United States. Performance Improvement Quarterly, 13(3), 84-109.
• Van Buren, M. E., & Erskine, W. (2002). The 2002 ASTD state of the industry report. Alexandra, VA: American Society of Training and Development.
• Whitfield, K. (2000). High-performance workplaces, training, and the distribution of skills.
Industrial Relations, 39(1), 1-25.
• Wright, P. M., & Boswell, W. R. (2002). Desegregating HRM: A review and synthesis of micro and macro human resource management research. Journal of Management, 28(3), 247-276.
• Wright, P. M., Gardner, T. M., Moynihan, L. M., & Allen, M. R. (2005). The relationship between HR practices and firm performance: Examining causal order. Personnel Psychology, 58(2), 409-446.
