The hypotheses in this study aimed at testing the moderation effects of important
moderators on the relationship between feedback variables and variables like affect towards task, self and feedback system as well self-efficacy and attribution that were also hypothesized as mediators in the relationship to other outcomes; these mediators are important outcomes in and of themselves and as possible mediators to other final outcomes not discussed in this study which is why moderation and mediation analyses were run separately above. However, it is also
important to realize that these variables, as mediators, need to affect the final outcomes studied here whether or not important moderators are included in the picture. Sometimes combining significant moderation with significant mediation yields insignificant mediation results (e.g. see Edwards & Lambert, 2007).
Thus, and because moderated mediation analyses cannot be run for all the combinations of predictors, mediators, moderators and outcomes in this study, these analyses were only run here using the Process macro in SPSS as explained earlier to ensure that the mediation results that are found and are presented as significant in this study remain so when combining them together with the moderators that were found to be significant in moderating the relationship
between the independent variables of this study and the hypothesized mediators; also the moderators that were significant only in coefficient but not in terms of R square increase after adding interaction were tested because as explained earlier sometimes combining moderation with mediation can change the effects and their significance (and thus, there is the possibility here that those marginally significant moderators may become fully significant). Also, when running moderated mediation using Process, trend and initial valence as explained earlier had to be translated into scalar variables which was a good opportunity to see whether initial valence and trend as scalar variables would yield the same results and shape of moderation effects as when they are tested as multi-categorical variables or not.
The moderated mediation results are displayed in appendix I in tables I-1a to I-1c (where table I-1a presents the results pertaining to initial valence as predictor, table I-1b presents results pertaining to trend as predictor and table I-1c presents results for feedback inconsistency as predictor). In these tables, the moderated mediation index as calculated by Process is displayed if significant; otherwise, whether it can be concluded that there is mediation without moderated mediation or no mediation at all is presented for the different mediators and moderators that were found significant in previous mediation and moderation analyses (see above) as well as the different outcomes hypothesized about in this study.
Not all moderated mediation results were significant so NA Feedback system does not mediate the relationship between inconsistency on one hand and desired, expected and standard grade on the other and NA Self does not mediate the relationship between trend and task
satisfaction or motivation (when control variables like general affect are accounted for). It is possible that with the inclusion of other moderators, mediation would again become significant but this is left here for future research. For all other cases, where there is significant moderation (at least in terms of coefficient) and mediation, there is either only mediation or moderated mediation (with a significant moderated mediation index); the details of direct effects and the indirect effects at different levels of the moderators are included in tables I-2a to I-2c.
Looking at the results for the moderated mediation in stage 1 for initial as predictor, the index for moderated mediation is only significant implying both mediation and moderation for those moderators that were associated with both a significant coefficient and significant increase in R square but not with those only with a significant coefficient like task difficulty manipulated in the case of relationship with PA Task for instance. In stage 5, in the case of trend, some of the
moderators like task importance (manipulated) that was only significant marginally (in terms of coefficient but not in terms of increase in R square after adding interaction) in affecting the relationship between trend and the mediator of PA Self led to significant moderated mediation for all outcomes except for learning intentions (however, PAVGO and Task importance
(measured) led as moderators between trend and PA Self when tested in a moderated mediation model led to insignificant moderated mediation indices as expected given their marginal
significance). Also, in the case of inconsistency, even though trend valence was found to be a significant moderator in the relationship between feedback inconsistency and internal attribution, a significant mediator for all outcomes except for performance criteria, testing a model of
moderated mediation showed none (only significant mediation) in the case of task performance satisfaction as outcome. Thus, looking at moderated mediation can change the significance of effects in this model but further exploration will be left for future research.
The Edwards and Lambert (2007) approach is used in tables I-3a to I-5b and figures I-1a to I-3b in appendix I (with the moderated mediation analyses and the calculated total direct, indirect and total effects separately shown in the tables I-3a to I-3c while the figures each contains all three effects as well as a diagram of the moderation of the predictor to mediator relationship for comparison with indirect effects if needed) to demonstrate three examples of moderated mediation. There are seven examples all in all, with two examples for significant moderation mediation with insignificant moderation and these two examples include one where initial valence is the predictor, PA Task stage 1 the mediator, task difficulty (measured) is the moderator and desired grade is the outcome. The other example is one where trend is the predictor, PA Self stage 5 the mediator, internal attribution is the moderator and learning intentions are the outcome
In the first example (see tables I-3c and I-3d and figure I-1b), the total effect means that for negative valence only, as task difficulty increases so does desired grade but very slightly while the opposite occurs for neutral and negative initial valence (again very small effects in terms of size). This total effect can be broken down into a direct effect whereby desired grade increases as initial valence increases from negative to positive, and an indirect effect which shows a sharp decrease in desired grade as task difficulty increases in the case of positive initial valence with an almost constant (or very slightly increasing) desired grade as task difficulty
increases for neutral valence and an increase in desired grade as task difficulty increases for negative initial valence.
In the second example, (see tables I-4e and I-4f and figure I-2c), the total effect means that for all three valences only, learning intentions increase as internal attribution increases but the extent to which the outcome increases as attribution to internal reasons increase is highest for positive trend followed by neutral trend followed finally by negative trend. This total effect can be broken down into a direct effect whereby learning intentions increase as trend valence increases from negative to positive, and an indirect effect which shows an increase in learning intentions as internal attribution increases but with this increase being steepest for positive trend followed by neutral trend and finally negative trend. The same logic applies to the rest of the cases.
Also, at least one example from each type of index: positive and significant, negative and significant, and non-significant is given with separate diagrams for total, direct and indirect effects. In each of the figure in appendix I, there are two or three versions of the indirect and total effects and this is because each one version would have the moderator on the X-axis to make it comparable to the moderation effect previously demonstrated (with a copy of the moderation effect figure demonstrated in appendix H again included in appendix I after the diagrams portraying the indirect effects) and one version would show the predictor variable on the X-axis; also, there is sometimes a third version of the diagram that shows the effect with the Y-axis scaled to start at zero (so that effect size is seen more clearly) but this version of each of the indirect and total effects is only shown when the effect is very small it requires a Y-axis not starting at zero to see. As seen in almost all of the figures presented in the appendix, regardless of significance or direction of moderated mediation index (which denotes the slope when mediated effects is drawn on the Y-axis with levels of the moderator on the X-axis; see Hayes, 2013), the relative indirect effects look almost exactly the same (in terms of overall direction and shape of the lines) as the related moderated effects previously displayed in appendix H whether this moderation effect was significant or not.
183 Chapter Ⅴ: Discussion
The structure of this section resonates with that of the results section. First direct relationship analyses results are summarized for both studies and discussed with possible explanations for discrepancies. The analysis of results will take the form of the suggestion of possible explanations for the important results as well as the insignificant results and the results that were different from what was hypothesized in the hypotheses with respect to the direct relationships in light of extant literature and theory. The results that were as expected will only be briefly discussed for length constraints. For this discussion, outcomes will be grouped into three groups with one including learning intentions/behavior, improvement intentions and motivation, a second containing satisfaction-related outcomes, and a third containing goals, expectations and standards. Afterwards, there will be a summary of results and short discussion for attention focus, and the team level relationships between primary outcomes and performance followed by the results for other important but supplementary findings in both the SONA study and BUSA study data. Finally, following the discussion of the results for direct relationships, the mediation and moderation (along with moderated mediation) analyses results will be presented and discussed for all the outcomes together.
Learning and Improvement Intentions and Motivation
In the SONA study, learning intentions as an outcome was not affected by either negative or positive initial valence (so added nothing to the average of learning intentions when initial valence is neutral) as opposed to what was found in some of the literature (e.g. see Kluger & DeNisi, 1996)). However, positive trend did have a positive effect over and above that of both neutral and negative trend while high inconsistency compounded that effect (strengthened it compared to consistency so had a positive effect) in stage 5 even after controlling for initial valence and overall mean. However, the effects that positive trend and inconsistency had on learning intentions in the SONA study did not withstand the addition of important control variables to the model. For instance, in this case, when controlling for conscientiousness (and other big five characteristics) the positive effects were rendered not significant. In other words, differences in conscientiousness may be driving the effects (or at least moderates them) observed in stage 5 with respect to learning intentions.
The BUSA study results mainly reinforced the results found in the SONA study with respect to all predictors: current (instead of initial) feedback (both value and valence), trend and
184
inconsistency of feedback (again both value and valence). So even though trend (value but not valence) did have a positive effect on learning behavior (only when there is a lag so trend at stage 2 tested with learning behavior at stage 3), this effect was very small; no other variables had a significant effect. Also, learning effort was not found to be a moderator of the relationship between current feedback value or trend and learning intentions even though it was found to have a direct positive effect. Finally, neither trend valence nor level of inconsistency when used instead of values affected learning intentions. Thus, generally it can be concluded that learning intentions are not hardly (if at all) affect by feedback variables; however, it is important to remember that learning behavior in the BUSA study was past behavior oriented while in the SONA study it was future behavior-oriented (i.e. more about intentions to behave than past behavior).
With respect to the outcome of general improvement intentions, very similar results to those of specific task-related learning intentions can be deduced from the SONA study results; so initial valence is not significant in stage 1 but positive trend had a positive effect. Inconsistency strengthened that effect (because it has a separate positive effect on improvement intentions). However, the effect of trend and inconsistency are only significant in terms of change in R square when important control variables like demographics are controlled for (but not when other control variables are controlled for). Thus, there is more promise in the direct relationship between trend and inconsistency on one hand and improvement intentions on the other than in the case of learning behavior and intentions where adding control variables renders all feedback variables insignificant.
In the BUSA study, however, trend (as it increases in value) was found to have a negative effect on improvement intentions rather than the hypothesized positive effect, i.e. as trend
increased, improvement intentions decreased but by a very small amount and only with control variables like level of effort and demographics added to the model. Also, current feedback valence only (positive valence had a negative effect while negative valence also had a positive effect as hypothesized but only when this valence was based on group-centered values and not when they are based on grand-centered values) but neither current feedback value nor the
valence of trend or value and levels of inconsistency had any effects on improvement intentions. In the case of motivation, the SONA study results show that negative initial valence increased motivation compared to neutral valence but the same was not true of positive valence
which did not significantly differ from neutral valence, contrary to what was predicted which was that both valences would increase motivation but negative valence would have a stronger effect. This effect of initial valence on motivation was only significant with control variables like demographics included in the model (but not generally or with other control variables like
general affect). On the other hand, contrary to what was hypothesized, trend up to stage 5 had no effect on motivation in stage 5 while feedback inconsistency only had an effect, which was positive contrary to what was hypothesized, when demographics are controlled for.
The BUSA study results show that when group-centering is used, trend, as it increases in value, can have a minute negative influence on motivation. Also, current feedback value has a small negative rather than positive effect on motivation (or as it increases, motivation decreases; also, when valence instead of value was tested, positive valence had a negative effect) and a positive effect when negative contrary to what was hypothesized (i.e. that both valences would increase motivation but negative valence has a stronger effect). Moreover, there is some (but little and needs further exploration) evidence of a lagging negative effect of initial valence on motivation in the second stage in the BUSA study after controlling for other variables like mean for rounds 2 to 5, trend and inconsistency values. This gives a preliminary answer to the question posed in the dissertation about lingering effects but mechanisms and reasons would need to be explore in future research; one possible explanation here can be that performance in the BUSA game builds on past rounds and so the effects of the first incident of feedback carries on in later stages. Another is as suggested by the literature on first impressions and resistance to
information that contradicts them regardless of evidence to the contrary as previously discussed (e.g. Tetlock, 1983).
Moreover, when trend valence rather than trend value is used as predictor, positive trend valence had a negative effect on motivation and the opposite was true of negative trend valence when group-centering is used (there are no significant effects when grand-centering is used), thereby again contradicting what was hypothesized. On the other hand, the extent to which inconsistency was generally higher or lower than group or grand average and its level in general had no effect on motivation in the BUSA study, further contradicting predictions.
These results uncover a small (if even significant) and unstable in between the two studies (and with the inclusion of control variables) effect of feedback on learning, improvement and motivation which contradicts many of the findings and predictions about the important role
of feedback in shaping these factors in the literature. One explanation for the often small and tentative effects (if any) of feedback on all three outcomes, learning, improvement intentions and task motivation, can be due to the argument in the literature about feedback only being effective and so play its role as expected with respect to outcomes when it is clear (e.g. Earley et al., 1990), and helps people understand what they did wrong and how to improve performance over time (e.g. see Hattie & Timperley, 2007); otherwise, it may be ignored (Ilgen et al., 1979), especially with little utility for meeting the performance standards and expectations set (see Klein, 1989). This is true of the SONA test. On the other hand, in the case of the BUSA game, even though more feedback and information on performance is given on a weekly basis to all teams (in the form of comparative indices and industry reports), there is little time to process this information especially in a team-based context. This can also explain the lack of a moderating effect for learning effort. Also, since in both studies, feedback is designed into the task, the job characteristics model (JCM) can apply here. According to JCM, feedback provision is related more to attitudes like jobs satisfaction than to work motivation (Fried & Ferris, 1987). The complete lack of effect on learning intentions and (very small effect on) behavior could also be due how these outcomes were measured.
Looking at the results for improvement intentions and motivation more specifically, in the case of general improvement intentions, initial valence did not affect them while there is evidence that positive trend increases them (and also learning intentions without control
variables in the picture and so similar explanations apply in this case) but overall valence (which also largely determines current feedback valence and not just based on initial valence as
previously discussed) has no additional effect in the SONA study. However, before discussing possible explanations for this effect, it is important to understand that this effect on improvement intentions and the effect of inconsistency on motivation discussed next become more pronounced only when demographics are accounted for (and can disappear completely with other control variables like general affect); most probably to the effect of factors like maturity (i.e. age and experience) on improvement intentions and motivation regardless of task at hand.
In general, however, these effects can be explained by the work on self-efficacy and the