Introduction
This chapter represents the ‘effect evaluation’ phase of the process evaluation “where data are analysed to determine the effects of the programme” (Nielsen & Abildgaard, 2013, p. 11). The main function of this chapter is to assess whether and where the interventions had any effect on employees’ perceptions of the psychosocial environment. That is, were there changes to psychosocial conditions and/or well-being-related outcomes, and did those changes correspond to those psychosocial conditions targeted by the interventions? In this case, the latter is addressed in two ways, by looking at both employees’ exposure to interventions and their perceptions of them.
The ‘screening’ phase (chapter five) considered data from the full T1 survey (n = 1,425) to determine the main psychosocial stress-risk factors, while the analyses in this chapter focus on participants whose baseline and follow-up surveys could be linked together (i.e. the repeated-measures sample). Surveys were linked using the anonymised self-generated identification code questions on each survey, described in the methodology chapter, which yielded a total repeated-measures sample of 552. This represented 38.7% of the original 1,425 respondents at T1, and 54.8% of the 1,008 who completed a survey at T2 (see appendix G for descriptive statistics from the full sample of the T2 survey; p.334).
The analysis described in this chapter progresses in three main stages. The first section simply looks at if there were changes to outcome variables across this sample between T1 and T2. This also includes an overview of qualitative data collected at T2.
189
However, it is the subsequent repeated-measures ANOVA’s that represent the main feature, because looking beyond simple T1 versus T2 comparisons, it was anticipated that exposure to interventions and perceptions of them would moderate changes in the psychosocial conditions targeted by each intervention component (e.g. Randall et al., 2005). Due to the timescale of the project and previous research indicating that changes to distal outcome variables (e.g. psychological health) tend to take longer to manifest themselves, and lag behind improvements to psychosocial environment (e.g. Dollard & Gordon, 2014; Wall & Clegg, 1981) the focus is on the latter and particularly the variables that were ‘targeted’ by the interventions.
• Effect evaluation aim 2: The effect of employees exposure to interventions and their perceived quality
Because of the importance of context in interpreting outcomes, the final element of the quantitative analysis explores employees’ pre-existing perceptions of the psychosocial environment (particularly T1 levels of CAOC), and its relationship to employees’ subsequent involvement (i.e. exposure) and rating of intervention activity. Although these are pre-existing factors and as such relate to the ‘initiation’ phase, these analyses are based on the linked dataset of 552 and process evaluation data collected at T2, (i.e. exposure & rating), and therefore it made sense to report these analyses together. The chapter concludes with a discussion of potential intervention effects (based on interaction effects from statistical analyses).
• Effect evaluation aim 3: assess relationships between employees’ baseline levels of cynicism about organisational changes and employees’ engagement with intervention activity
190
Data screening
The analysis of pre- and post-intervention surveys here is based on the repeated measures sample of 552, and pre-analysis screening of data for the repeated-measures t-tests confirmed the univariate normality of the variables for the first analysis. Histograms suggested moderate skew in a small number of variables, but skewness and kurtosis statistics for all variables were within acceptable limits (i.e. skewness between 2 & -2, kurtosis between 7 & -7; West, Finch, & Curran, 1995). Skewness for all variables at both time points ranged from 1.11 (GHQ-12 at T2) to -0.74 (control at T2), with kurtosis ranging from 0.98 (GHQ-12 at T2) to -0.50 (change at T2).
The suitability of the data for repeated-measures ANOVA, based on the assumption of homogeneity of variances, was assessed using guidelines from Baguley (2012). Levene’s test is oversensitive in large samples, such as the present study, so it is recommended that the standard deviation of dependent variables for each level of the between-group factor should be no more than twice as large as another (Baguley, 2012); in all cases they were within this threshold and so met the criteria for homogeneity of variance.
Because the repeated measures sample represents only a proportion of employees who completed the first survey, the potential effects of participant attrition and the possibility of differences between T1-only and repeated measures sample were assessed. Independent t- tests were conducted on all study variables, as well as gender, age, and full/part-time status with survey participation coded as 1 = both surveys, 0 = T1-only. Control was the only study variable that differed significantly between the groups: t(1420)=4.19, p<.001, with ‘T1 only’ (n = 867) participants having lower levels of control (mean = 3.45; SD = 0.77) than the 552 employees who participated in both surveys (mean = 3.62; SD = 0.70). See appendix E (p.332) for t-tests of T1-only participants versus staff completing both surveys.
191
The same analysis was conducted on T2 variables, comparing those who completed both surveys with those only completing T210. There were several significant differences between T2-only surveys and those completing both (at p < .01), with a consistent pattern of more positive scores for T2-only participants, for demands, role clarity, change, POS, and job insecurity. The one exception was control, which was higher (better) for employees who completed both surveys. The T2-only participants were also more likely to be full-time (see appendix F, for mean scores for participants completing T2-only versus those completing both surveys; p.333). As a consequence, independent t-tests comparing the full T1 and T2 samples are slightly more positive than the repeated measures t-tests shown in table 19, on the next page, with only job insecurity and JRWB (both improved) showing a significant change (appendix G for t-test summary table).
Only a selection of the variables measured here were targeted by PublicOrg’s interventions, as described in the previous chapter, but the tables summarising findings include results for all of the main study variables. On the one hand, this provides transparency, but it also allows us to rule out (or not) the possibility that any statistically significant effects may be an intervention-induced Hawthorne/‘halo-effect’ (Sørensen & Holman, 2014); while changes to perceptions of psychosocial conditions can provide some evidence for intervention efficacy, the assessment of whether observed changes correspond to those conditions targeted by intervention(s) is also important to discern (Nielsen & Abildgaard, 2013). Evidence for this is strengthened when these changes are accompanied by a lack of change in ‘irrelevant’ variables (i.e. those not targeted by interventions).
10 This is based on employees who completed a T2 survey that could not be linked to a T1 survey using the
self-generation identification code. Therefore it is possible that some participants categorised as T2-only could have completed both surveys but could not be matched due to insufficient (or incorrect) linking code information.