In order to determine the quality of stMLE parameter estimates, compared to nMLR, a Monte Carlo simulation study was conducted. Two population models were specified: ‘Linear Growth’ and ‘Zero Growth’ models. For both models, population values were based in part on parameter estimates from the empirical study discussed above. Data was generated based on a linear LGM, with five time points, and
heteroscedastic and independent residuals. The “Linear Growth” population data was generated based on the following matrices:
Σ = ΛΦΛ'+Θ; α1 = 60; α2 = 3; 𝛷 = [40 −5 −5 2 ] 𝛩 = [ 40 0 0 0 0 0 35 0 0 0 0 0 30 0 0 0 0 0 25 0 0 0 0 0 15]
As for the zero-growth model, the same matrices above were used; however, the slope factor mean and the correlation between intercept and slope were set to zero. Data was generated using the package SIMSEM in the statistical software R
(Pornprasertmanit, Miller & Schoemann, 2016). Simulation Conditions
Five simulation factors were manipulated: population model, sample size, distribution of latent variables, percentage of attrition/missingness, and estimation
28
procedure (Table 5). The two population models are described above, and were chosen so that both power and Type I error for each estimation procedure can be examined. Model specification was fixed across all conditions. Five hundred replications were analyzed per condition. Overall, there were 72 simulation conditions, or 36 conditions per estimation procedure (18,000 replications total).
Sample Size. Sample sizes were set to 100, 500, and 1000. The minimum was chosen as 100 since most SEM models are estimated with 100+ observations (Jackson et al., 2009) and SEM is considered a large samples approach (Kline, 2015). Additionally, the max sample size was 1000 because stMLE may require larger sample size to
converge, especially when there are missing data, and 1000 would be considered large in several educational research fields.
Distribution. The degree of non-normality of the endogenous variables was manipulated indirectly by specifying the distribution of all latent variables (intercept factor, slope factor, and residuals). Specifically, a sequential method generates factor data first, based on specified distributions for the factors, then a set of equations is applied to obtain the observed data, following the Vale and Maurelli (1983) method (Pornprasertmanit et al., 2016). With that approach, if all latent variables are normally distributed, then the endogenous variables will in turn also be normally distributed, since they are the sum of a linear combination of latent variables.
The distribution of the latent variables was manipulated into three conditions: normal, moderate non-normality, and severe non-normality. Values for univariate skewness and Kurtosis for latent variables are based on Shin et al. (2009) and Cain,
29
Zhang, and Yuan (2017) review of typical univariate skewness and kurtosis values in real data. The following are the values used for different distributional conditions of all latent variables:
Normal: skewness = 0 and kurtosis = 3
Moderate non-normality: skewness = -3.25 and kurtosis = 5 Severe non-normality: skewness = -5 and kurtosis = 10
Missingness. In every longitudinal study, it is likely that some individuals will drop out of the study. All missingness in this study was assumed to be missing
completely at random (MCAR). There were two missingness conditions specified: complete data and 10% attrition. For both conditions, the first time point does not have any missingness. Missingness was created by dropping 10% of the cases at each time point. Once a case has been dropped, it no longer had any data for the subsequent time points.
Estimation. Linear growth model was specified for all conditions, including the “Linear Growth” and the “No Growth” models. All models were correctly specified, and misspecification was not considered. All data was analyzed with two estimation
approaches: nMLR and stMLE. All analyses are conducted in Mplus using the Monte Carlo option with population parameters specified as starting values12. Extraction of
simulation results were carried out in R, using the package” MplusAutomation” (Hallquist & Wiley, 2014).
12 With the exception of distributional parameters (df and skewness) because specifying
30 Evaluation Criteria
The performance of nMLR and stMLE was evaluated based convergence rates, relative bias, mean square error (MSE), 95% coverage, and Type I/Power. The accuracy of parameter estimates was evaluated through relative bias and MSE, and the
performance of standard errors was evaluated with 95%coverage rates, Type/Power. First, convergence rate was determined by the proportion of replications that resulted in successful computation, with or without warnings13. Successful computations include those replications with warning messages. None of the successful computations with warning messages were excluded in order to maintain a sufficiently high number that is representative of replications across all conditions, an approach taken by other SEM researchers (e.g. Leite, 2007; Ulitzsch et al., 2017).
Second, relative bias refers to the difference between the averaged parameter estimates within each condition and the true population parameter (Ender's, 2001). Relative bias was calculated by the following formula (Muthén & Muthén, 2016):
𝑅𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝐵𝑖𝑎𝑠 =(𝜃̂ − 𝜃)
𝜃 ∗ 100
where 𝜃̂ is average of parameter estimates, and 𝜃 is the population value. If the value of relative bias is below 15%, it is considered acceptable (Muthén, Kaplan, & Hollis, 1987).
31
Third, MSE is interpreted as a combination of bias and sampling variability. It represents the overall accuracy of an estimate (Enders, 2001; Muthén & Muthén, 2016). When the parameter estimate is unbiased (absolute bias14 = 0), MSE represents
efficiency, or the sampling variability, of a parameter estimate. MSE values for the same parameter from different conditions can be compared to determine the relative efficiency of estimation procedures. MSE is calculated as (Muthén & Muthén, 2016):
𝑀𝑆𝐸(𝜃̂𝑗) = 1
𝑁 − 1 ∑[𝜃̂𝑖𝑗 − 𝜃̂𝑗]
2 𝑁
𝑖=1
where 𝜃̂𝑖𝑗 is the estimated parameter for sample i, 𝜃̂𝑗 is the specified population value for
parameter j, and N is the number of replications.
Fourth, coverage refers to the proportion of 95% confidence intervals (CI) that contained the true population value for each parameter (Muthén & Muthén, 2016). Coverage provides a measure of how well the parameters and their standard errors are estimated. It is expected that coverage values would be close to 0.95%. Lastly, power was determined for all parameters in the “linear Growth” conditions, and Type I error was determined for the Slope and Slope/Intercept covariance parameters for the “No Growth “conditions. Power is the probability of rejecting the null hypothesis when it is false, and is expected to be approximately 0.8. Type I error is, the probability of rejecting the null hypothesis when it is true, is expected to be below 0.05 (Muthén & Muthén, 2016).
32
5. RESULTS: MONTE CARLO SIMULATION