The repeated measures ANOVA carried out on the data from the control group revealed a significant effect of time on all of the study measures: self-efficacy
F (2, 10) = 6.40, p <.05; developmental support F (2, 10) = 8.75, p <.01; positive
affect F (2, 10) = 11.58, p <.01; openness to new behaviours F (2, 10) = 29.83, p <.01; and developmental planning F (2, 10) = 54.25, p <.01 (refer Table 4.8 on page 122). The effect size, measuring the magnitude of a treatment effect independent of sample size, was calculated for the variables (Lipsey & Wilson, 1993). There are a number of indices that measure effect size, in a repeated measures ANOVA the most commonly used measures to calculate effect size are Eta squared (η2
) and partial Eta squared (ηp2) (Bakeman, 2005; Levine & Hullett, 2002), which has been reported in
Table 4.8, on page 122. Eta squared and partial Eta squared are estimates of the degree of association for the variables (Levine et al., 2002).
A Pairwise comparison, with a Sidak adjustment to control for familywise Type 1 error, was used for the post-hoc analyses to examine the nature of the time effect. These analyses revealed that participants’ feelings of self-efficacy increased significantly pre-coaching (wave 2) to the follow-up six months after completing executive coaching (wave 5), mean difference .54, p < .05, and developmental support from pre-coaching (wave 2) to the follow-up six months after the last
executive coaching session (wave 5), mean difference 1.3, p <.05. Positive affect increased from pre-coaching (wave 2) to the follow-up six months after completing executive coaching (wave 5), mean difference 1.05, p <.05, and also significantly increased from after six coaching sessions (wave 3) to the follow-up six months after completing executive coaching (wave 5), mean difference .50, p <.10. Openness to new behaviours demonstrated significant increases pre-coaching (wave 2) to after
six executive coaching sessions (wave 3), mean difference 1.13, p <.05, and pre- coaching (wave 2) to the follow-up six months after the final executive coaching session (wave 5), mean difference 1.3, p <.05. Developmental planning also demonstrated significant increases from pre-coaching (wave 2) to after six executive coaching sessions (wave 3), mean difference 1.47, p <.05, and pre-coaching (wave 2) to the follow-up six months after completing executive coaching (wave 5), mean difference .16, p <.05. Thus, when the analysis was focused on the data collected from the control group, the data were more consistent with Hypothesis 2, although not all variables demonstrated a significant increase from pre-coaching (wave 1) to after six coaching sessions (wave 3). However, the lack of findings could be
attributed to low statistical power. A power analysis, based on Cohen (1988) using a G Power Test (Faul & Erdfelder, 1992) suggested that a sample of 210 participants would be required for this effect to be significant for self-efficacy and developmental support (which only demonstrated significant change from pre coaching (wave 2) to 6 months after completing executive coaching (wave 5)). The power analysis also indicated that there was only a 19% chance of getting a significant effect with this small sample from pre-coaching (wave 2) to six months after completing executive coaching (wave 5) (Faul et al., 1992), yet this pattern of results was demonstrated for all psychological variables. Therefore, the key question underlying the longitudinal analysis: whether the psychological effects associated with executive coaching would be maintained six months after regular coaching sessions had ended, was answered. The post-hoc analyses indeed demonstrated that all measures remained above their pre-coaching level when they were measured at the six month follow-up. The pattern of change over time on these variables (for the control group) is presented graphically on pages 122 to 124 (see Figure 4.12 to Figure 4.16).
Table 4.8
Longitudinal Effects of Executive Coaching on the Psychological Measures: Means, Standard Deviations and ANOVA Statistics – Control Group dataa
Pre- coaching (Wave 2) After six coaching sessions (Wave 3) Six month follow-up (Wave 5) Variable
Mean s.d. Mean s.d. Mean s.d. F df ηp
2 Self-efficacy 3.72 .57 3.82 .77 4.26 .69 6.40* (2, 10) .56 Developmental Support 3.27 .59 3.80 1.07 4.53 .40 8.75** (2, 10) .63 Positive Affect 3.35 .49 3.90 .58 4.40 .51 11.58** (2, 10) .70 Openness to New Behaviours 2.92 .47 4.04 .51 4.21 .51 29.83** (2, 10) .86 Developmental Planning 2.67 .84 4.13 .41 4.30 .55 54.25** (2, 10) .92 a n=6. * p < .05 ** p < .01 Two tailed tests.
Self-efficacy
1 2 3 4 5Pre EC After 6 EC sessions 6 months after EC
Time
Mea
n
Figure 4.12. Longitudinal Effects of Executive Coaching on Self-efficacy (Control
Developmental Support
1 2 3 4 5Pre EC After 6 EC sessions 6 months after EC
Time
Me
an
Figure 4.13. Longitudinal Effects of Executive Coaching on Developmental Support
(Control Group).
Positive Affect
1 2 3 4 5Pre EC After 6 EC sessions 6 months after EC
Time
Mea
n
Figure 4.14. Longitudinal Effects of Executive Coaching on Positive Affect (Control
Openness to New Behaviours
1 2 3 4 5Pre EC After 6 EC sessions 6 months after EC
Time
Mea
n
Figure 4.15. Longitudinal Effects of Executive Coaching on Openness to New
Behaviours (Control Group).
Developmental Planning
1 2 3 4 5Pre EC After 6 EC sessions 6 months after EC
Time
Mea
n
Figure 4.16. Longitudinal Effects of Executive Coaching on Developmental
Discussion
The aim of this study was to establish the effects of executive coaching on leaders’ psychological states, specifically, their self-efficacy, developmental support, positive affect, openness to new behaviours and developmental planning. Two approaches were used to investigate these effects. First, an experimental design was used to establish the impact of executive coaching on leaders’ psychological states. Second, longitudinal analyses were used to examine whether the effects of executive coaching were sustained over time. The results of both analyses provide support for the study hypotheses, though they must be interpreted with some caution, due to the small sample size.
In the first analysis, the training group who had received six sessions of executive coaching, reported significantly higher levels of self-efficacy,
developmental support, openness to new behaviours, and developmental planning compared with the control group, although no significant effects were observed for positive affect. However, the significant differences between the training group and the control group occurred because the control group decreased (from the first pre- coaching measure – wave 1, to the second measure collected after the training group had completed six executive coaching sessions, and immediately prior to the control group starting coaching – wave 2), rather than because the training group showed an increase on the psychological measures. This differed from the effect expected as a result of the training group receiving executive coaching.
The design of this study, wherein leaders were randomly assigned to
choice to help distinguish the effects of executive coaching from other
developmental interventions (e.g. 360 degree feedback and the two day training workshop) which can cause differentiation, and subsequent interpretation difficulties (e.g. Thach, 2002). At this stage in the data collection process, there were no
significant differences between the training and control groups although, notably, with a small sample it is impossible to control entirely for factors like age,
experience and level of responsibility. However, an examination of the demographic characteristics suggested the leaders were evenly spread through random assignment. Therefore, it was expected that any differences between the training and control group at wave 2 (after the training group had received executive coaching, but before the control group commenced) could be attributed to the effects of executive
coaching. However, the pattern of results measuring the differences between the two groups at wave 2 did not demonstrate this outcome. Even though there was a
significant difference at wave 2 for all variables except positive affect, this prediction is difficult to support because the post-hoc analyses revealed that it was the control group, rather than the training group, which showed the most change on the study measures from wave 1 to wave 2. This is despite the fact that the control group did not experience any intervention during this period. Further confounding the
difficulties interpreting the results, the pattern of significant change for the control group represented a decrease rather than an increase on measures of self-efficacy, developmental support and developmental planning. There was no significant decrease in the control group noted for openness to new behaviours, although there was equally no significant increase reported by the training group. Developmental planning did significantly increase for the training group between wave 1 and wave
2, but a significant decrease in the control group ratings was also noted. There were no significant effects for positive affect.
There are two possible ways to interpret these unexpected results. First, the decrease from wave 1 to wave 2 for the control group could be due to resentful demoralisation, which is an issue in controlled experiments in which participants in the control group become resentful of not receiving the experimental treatment (Cook et al., 1979). These participants may become angry, depressed,
uncooperative, or non-compliant. This may lead to significant systematic differences in the outcome of the control group, whereby a post-test difference between the treatment and non treatment groups occurs. This obscures the results of the study and threatens its validity when the difference between groups is attributed to the planned treatment, yet in fact it is actually due to inadvertent resentful demoralisation experienced by the control group (Cook et al., 1979). Resentful demoralisation, however, is most likely to occur when the experiment is obtrusive and the control group receives no treatment or less desirable treatment (Cook et al., 1979). This situation does not accurately describe the management of the control group within this study. Participants in the control group knew that they were to receive executive coaching within approximately twelve weeks of the training group commencing their coaching sessions. Anecdotal feedback from control group participants did not support the notion that they were angered or depressed with waiting. It was quite the opposite actually, with some training group respondents inquiring as to whether they could move into the control group; and leaders who were in the control group
evidence to rule out the fact that resentful demoralisation could be responsible for the decrease demonstrated.
An alternate explanation is that this pattern of results occurred because the pre-coaching measures were obtained at the end of the two day training workshop, when the psychological measures may have already been relatively high. If this is the correct interpretation, then the effect of executive coaching appears to have been to sustain the impact of the workshop. On the other hand, for the control group who did not experience follow-up through executive coaching immediately after the workshop, the psychological measures decreased to their normal levels pre the two day training workshop. There is support in the literature for this proposition. As discussed previously, researchers have reported a common immediate outcome of training is that that participants experience feelings of elation and leave with a moderate “buzz” from the energy and enthusiasm generated by the training (Alliger et al., 1997; Dearborn, 2002; Ruona et al., 2002; Warr et al., 1995). These feelings however, are unlikely to be sustained back in the workplace, and are therefore a criticism levelled against the short-term impact of training workshops (Dearborn, 2002). This explanation supports the proposition that the pre-coaching measures (wave 1) used as the baseline in this study, might have been inflated because they were collected at the end of the training workshop. Furthermore, the decrease on the study measures shown by the control group, who received no further intervention between wave 1 and wave 2, is also consistent with this suggestion.
This second explanation seems more accurate within the organisational setting in which executive coaching occurred. If indeed it does explain the
executive coaching, which took place in the context of an on-going training program, maintained the relatively high levels of self-efficacy, developmental support,
openness to new behaviours, and developmental planning generated by the two day training workshop. Despite the fact that this was not the anticipated effect predicted from executive coaching, it is still advantageous because it suggests that executive coaching offers a means of sustaining the positive effects of training over time. Further interpretation about the effect of executive coaching for the control group can be gained from the longitudinal analysis discussed shortly.
An alternate interpretation of the results was calculated with Cohen’s d, which measures the effect size or magnitude of a treatment effect, independent of sample size (Cohen, 1988). According to Cohen’s (1988) criteria, all of the effect sizes reported here comparing the post-coaching training group means with the post- coaching control group means (i.e. at wave 2) qualify as large (Cohen, 1988). The exception was positive affect which had a small effect size. These effect sizes support the explanation that executive coaching is impacting positively on the psychological measures and provide useful information for practitioners wishing to predict the effectiveness of executive coaching as a developmental intervention (Sigurdsson & Austin, 2007). Care must still be taken though generalising from small sample sizes due to sampling error inherent in such situations.
The longitudinal investigation was designed to ascertain whether the
observed effects of executive coaching would be sustained over time4. The results, (when restricted to the control group leaders whose pre-coaching measures were not as high as the training group) revealed significant change over time on all of the study measures, from prior to executive coaching, to six months after the completion of executive coaching. These results provide some support for the suggestion that the impact of executive coaching is sustained over time.
However, because it is not possible to confidently explain why the control group decreased from their first set of pre-coaching measures (wave 1) to the second set of control group pre-coaching measures collected (wave 2, before they received executive coaching), the longitudinal analysis using the combined sample from the training and control groups should also be considered. The combined results
revealed significant change over time on positive affect, openness to new behaviours and developmental planning from prior to executive coaching, to after six executive coaching sessions (openness to new behaviours and developmental planning); and from prior to executive coaching, to six months after the completion of executive coaching (positive affect and developmental planning). These analyses partially support the second hypothesis (with the exception of self-efficacy and developmental support), but with a less consistent pattern than that observed when only interpreting control group data.
4
In line with the design of this study, the control group had received executive coaching by this time and provided data for the investigation. The control group is therefore the second experimental group however they will continue to be referred to as the control group to facilitate identification.
The pattern of change on the control group data observed most consistently from the analyses was different from what was expected. It was expected that executive coaching would bring about an improvement on the study measures from prior to coaching to session six, and the interest was in determining whether the effect would be sustained over the six month period after regular coaching sessions ended. However, it was not expected that there would be any additional increase in the study measures from session six up to the six month follow-up. The most consistent effect observed in the data though was a significant increase from pre- coaching to the six month follow-up. In other words, there was a tendency for the measures to continue to increase in the period between completing the sixth session and the six month follow-up. Unlike traditional forms of training which do not seem to have a sustained impact (Conger et al., 2003; Warr et al., 1995), it may be that the effects of executive coaching are enhanced over time because the leaders are learning and practising techniques that they should continue to use after coaching finishes. However, because there was not a control group at the time of the six month follow- up, further testing is required before any firm conclusions about the long-term effects of executive coaching can be drawn. It is possible that other events occurring during this six month period (e.g. follow-up training workshops) contributed to the trend observed.
Although these findings must therefore be viewed as tentative, they are consistent with the idea that more long-term interventions such as coaching should improve developmental outcomes (Tobias, 1996). According to Falk (2003)
development should take place over a long period of time, with regular sessions. The findings of the current research support the need for leadership interventions to be
more than a “once-off” or short-term program, and the design of executive coaching appears to meet this need.
It is worth noting that the observed pattern of change over time on the control group data was not entirely consistent for the different dependent variables. The most immediate impact (a significant change from pre-coaching to session six) was observed for the measures of openness to new behaviours and developmental planning, suggesting that executive coaching had a fairly immediate effect on these measures. In contrast, the measures of self-efficacy, developmental support and positive affect increased more gradually, from pre-coaching to the follow-up six months after executive coaching. This pattern of change is consistent with existing research, which suggests that goal setting (a key element of developmental planning) is a precursor of motivation and job satisfaction (Locke & Latham, 1984; 1990b; 2002; Yearta, Maitlis, & Briner, 1995), and that creating openness to change is a precondition for bringing about other changes (Lewin, 1947). These initial effects of executive coaching may serve as precursors for subsequent effects on self-efficacy, developmental support and positive affect. Whilst sample size limitations preclude this analysis from being conducted within this study, it is a direction future research could examine.
The fact that the measures of self-efficacy, developmental support and positive affect did not show the expected change from pre-coaching to session six, and instead only showed an overall change from pre-coaching to six months after executive coaching may perhaps be attributed to the small sample size. The power analysis indicated that a much larger sample would be required for a significant effect to have a good likelihood of being demonstrated. However given the inherent
difficulties collecting executive coaching data and the substantial time commitment necessary for coaching each leader, collecting data from such a large sample was not feasible for this research program. The significant effects that were obtained from the study analyses are impressive given that the power analysis indicated that there was only a 19% chance of getting a significant effect with this small sample (Faul et al., 1992).
Although the results for the longitudinal analysis focusing on the control group have to be interpreted with caution due to the small sample size, it is reassuring that similar patterns of change were demonstrated for the combined training and control group sample. That is, the longitudinal analyses for the combined sample also demonstrated significant change from pre-coaching to after six executive coaching sessions for leaders’ openness to new behaviours and developmental planning. Significant differences were also reported from pre-
coaching to six months after the completion of executive coaching for positive affect and developmental planning.
However, as some of the effects confirming the impact of executive coaching over time were only demonstrated when the control group data was analysed, taking into account the very small sample, re-testing the longitudinal psychological effects associated with executive coaching would be beneficial. Indeed, although the difference in scores from the training and control groups has been attributed to inflated pre-coaching data as a consequence of the workshop, an alternative
explanation already offered is that having to wait for coaching actually deflated the second round of measures collected from the control group (i.e. the data collected immediately prior to the control group commencing executive coaching). To avoid
this situation again in future, research pre-coaching measures should be collected immediately prior to the leaders commencing executive coaching rather than at the