Statistical Analyses

Screening. Prior to data analysis, we conducted a screening procedure as suggested by Tabachnick and Fidell (2012), including checks of missing values, distribution assumptions, univariate and multivariate outliers, and multicolinearity.

Pre-Post Analyses. To test our hypotheses concerning the training effect, we conducted two mixed effects MANOVAs (i.e., doubly multivariate profile analysis; Tabachnick & Fidell, 2012). Each mixed effects MANOVA included the same independent variables but different dependent variables. The dependent variables in the first MANOVA were math achievement, SRL overall, SRL knowledge, and self-efficacy. This MANOVA was intended as a broad test of training effects in all targeted constructs. The dependent measures in the second MANOVA were the seven SRL- subscales goal-setting, planning, motivation, distraction avoidance, self-instruction, learning- strategies, and reflection. The goal of this analysis was to differentially analyze improvements in the individual SRL components. Our three dichotomous independent variables were SRL- training (yes/no), learning diary (yes/no), and time (pre- or post-training). This resulted in a 2 x 2 x 2 design, with time being the repeated measure. A significant time x training interaction would indicate a positive training effect. The effect sizes partial η2 _{were calculated as suggested} by Tabachnik and Fidell (2012). However, as they outline, partial η2 _{forMANOVAs may not be} interpreted as the proportion of variance explained in all DVs. Nevertheless, it still represents a measure of relative importance of the effect.

To clarify the multivariate results and determine which variables were dominantly affected by the training, we followed up on these analyses by conducting linear discriminant analyses (LDA) as suggested by Field, Miles, and Field (2013) as well as Tabachnick and Fidell (2012). In LDA, a number of factors (or discriminant functions) are extracted from the DVs that best discriminate between groups (i.e., the IVs). As we were primarily interested in 1) change in variables 2) caused by our training (the time x training interaction), we used the post-score – pre-score difference scores of our DVs as the basis for factor extraction. As groups, we only compared training vs. no-training groups to describe their differences as accurately as possible. Therefore, we estimated only one discriminant function. To produce LDA coefficients that are not influenced by the respective scale, we standardized all pre and post scores based on pre measurement statistics – e.g., we standardized the post math score using the mean and standard

deviation of the pre math score. This procedure led to equally scaled post – pre differences that reflected individual changes in pre-measurement standard deviations.

Time-Series Analyses. For time-series analyses, we included only the diary groups TD and D. We examined process aspects of SRL by calculating trend analyses and intervention analyses. To carry out trend analyses, the daily data of each participant was aggregated to a daily mean. For the resulting 27 days, a linear regression was modeled with time as the predictor and the value of the scale as the criterion. This can be done either at the individual level, which results in a trend for each person, or as a group aggregate. We chose the latter approach in order to investigate the mean training effect on group TD compared to control group D.

Intervention analyses (interrupted time series analysis) were calculated in order to examine whether an intervention has an effect on a system, how the intervention influences the system, and which other variables influence the dependent variable. The intervention effect is indicated, similar to a t-test, by the comparison of the baseline level of the dependent variable to the level after the intervention. A transfer function is used to estimate how the intervention influences the system. ARIMA models (Schmitz, 1990) serve to describe other sources of impact.

Traditionally, intervention analyses are calculated for the whole group by aggregating over persons (for a detailed description see Schmitz, 1990). This approach was not suitable for our study design because participants could choose the day on which they took part in a lesson. Aggregating over participants would therefore result in a loss of data or accuracy as some might have already completed a lesson at a given day while others might not have started yet. In cases of varying intervention onset for each individual, we propose another approach: by calculating an intervention analysis for each participant, each individual is treated as its own sample. All samples are then aggregated by means of meta-analytic methods.

This approach results in a t-value for each individual and each intervention, which can be converted to an effect size d following the formula

(1) & = ( ∗ *+,*-

*+∗*-

with n1 and n2 referring to the number of diary entries before respectively after the intervention. This effect size represents the impact a given intervention had on a given participant. Aggregating over participants is then done with meta-analytic techniques using a random- effects model. For this purpose each effect size is weighted by the inverse of the variance to give more weight to effects based on large samples. The variance is calculated as

(2) ._/ =(*+,*-)

(*+∗*-) +

34-

Computation of random effects models for meta-analysis is described in detail by Raudenbush (1994). We applied this procedure for each of the three lessons separately with both the mean SRL score and the three subscales for lesson 1 strategies, lesson 2 strategies, and lesson 3 strategies.

6.4 Results