Simulation Analysis

The one-size-fit-all cutoffs, the Bayesian approach, and the simulation approach provide only two possible outcomes: trivial or severe misspecifications. I refer to these methods as two-outcome methods. The unified approach, the tests of close and not close fit approach and the modification indices and power approach provide three outcomes: trivial misspecifications, severe misspecifica- tions, or inconclusive. I refer to these methods as three-outcome methods. For the three-outcome methods, two outcome variables are obtained: the proportion of inconclusive (or underpowered) results and the proportion of model rejection among conclusive results. If the proportion of incon- clusive results is greater than .90, the rejection rate would be computed based on a small proportion of conclusive results so I code the rejection rate as missing in this case. On the other hand, two- outcome methods will provide only the proportion of model rejection.

I mainly use two tables to evaluate the influence of the design conditions on the performance of the model evaluation methods. These tables present results from factorial analysis of variance (ANOVA). The design factors in this analysis include the size of target loadings, the number of in- dicators, sample size, the degree of misspecifications, the level of maximal trivial misspecification, and the type of misspecifications. However, all factors are not fully crossed. Type of misspecifica- tion is only applicable when the level of misspecification is not Level 0. Therefore, in the factorial ANOVA, I did not include the results with Level 0 degree of misspecification. The dependent vari- ables for the first and the second tables are the rejection rates and the proportions of inconclusive results, respectively. The proportions of inconclusive results are only applicable for the three-

outcome methods. Note that each combination of the design factors has only one observation of the dependent variables. Therefore, the highest-order factor cannot be separated from the error

variance. The factors with eta-squares (η2s) of .03 or higher are deemed non-negligible factors.

The past research has used .01 (Lüdtke et al., 2008) or .05 (Geldhof et al., in press) as threshold

for meaningful η2s. I found that, in the current study, .01 included factors with negligible effects

whereas .05 ignored important effects. Thus, I chose .03 as the cutoff.

When any main or interaction effect has a non-negligible effect, I describe the pattern of the effect. In addition, the averages of the dependent variable across each level of the non-negligible conditions are tabulated to further explain the effect. If an effect is related to the degree of mis- specification, the results associated with Level 0 degree of misspecification are included. Given that the interaction between the degree of misspecification and the level of trivial misspecification is the primary research question of the study. The tables of the averages of the rejction rates and the proportion of inconclusive results across the degree of misspecification and the level of trivial misspecification are provided even if the effects are negligible. If the interaction is dependent on other factors (i.e., three-way or higher interaction), the table describing the higher order interaction is shown instead. All tables mentioned above are used to answer the research questions as follows.

4.1.4.1 The Comparisons between Model Evaluation Methods

All model evaluation methods are evaluated in terms of to what extent they have satisfied the desired properties. The desired properties are as follows:

Appropriate Rejection Rates for Varying Degrees of Misspecification and Levels of Trivial Misspecification. The hypothesized model should be rejected if the model misspecification is higher than the maximal level of trivial misspecification. In contrast, the hypothesized model should be retained if the misspecification is lower than the maximal level of trivial misspecification. From the tables described above, four results are used to indicate whether this desired charac- teristic is satisfied. First, the degree of misspecification and the level of maximal trivial misspeci- fication should interactively influence the rejection rate. That is, a good model evaluation method

should have high η2 on the interaction between the degree of misspecification and the level of maximal trivial misspecification. Second, the rejection rates of trivial misspecification conditions should be close to 0. I consider the rejection rates lower than .10 as desirable. Third, the rejec- tion rates of severe misspecification conditions should be close to 1. I consider the rejection rates higher than .90 as desirable. Fourth, the rejection rates of the cutoff conditions (the model misspec- ification is at the same level as the maximal trivial misspecification) should be in between 0 and 1 because it is unclear to be considered the cutoff conditions as trivial or severe misspecification. I consider the rejection rates between .10 and .90 as desirable.

Rejection Rates Are Not Influenced by Types of Misspecification. There are three types of misspecification in this simulation: the misspecification in factor correlation, the misspecification in cross loadings, and the misspecification in error correlations. A good model evaluation method should be able to detect all types of severe misspecifications. Thus, a good model evaluation

method should have low η2s (< .03) on the main and interaction effects involving the type of

misspecification.

Rejection Rates Are Not Influenced by Model Characteristics. This simulation investigates two model characteristics: the number of items (8 and 16) and the magnitude of target factor loadings (0.5 and 0.7). The rejection rates should be consistent across the numbers of items and the magnitudes of target factor loadings. Thus, a good model evaluation method should have low

η2s (< .03) for the main and interaction effects involving the number of items or target factor

loadings.

Rejection Rates Are Not Influenced by Sample Sizes. A good model evaluation method should consistently retain trivially misspecified models and reject severely misspecified models across all sample sizes (125, 250, 500, 1000, 2000, and 4000). The effect of sample sizes on the proportion of inconsistent results is investigated in the next section. A good model evaluation

4.1.4.2 The Properties of the Unified Approach

Because the unified approach provides three possible outcomes and consists of both global and local fit evaluations, studying the pattern of the proportion of inconclusive results and the congru- ency between the global and local model evaluations would provide a better understanding of the performance of the approach.

Pattern of the Proportions of Inconclusive Results. The main benefit of the unified approach is that it does not provide the decision of model rejection (reject vs. retain a model) if it does not have enough information. There are two situations indicating low information:

1. Sample size is low.

2. The degree of misspecification is close to the level of maximal trivial misspecification (es- pecially in the cutoff conditions).

Therefore, the η2s on the proportion of inconclusive results should be high for the main effect

of sample size and the interaction effect between the degree of misspecification and the level of maximal trivial misspecification. Furthermore, the table of the average proportion of inconclusive results classified by the degree of misspecification and the level of maximal trivial misspecification will be shown. The proportion of inconclusive results is expected to be the highest when the sample size is smallest holding the other factors constant and at the cutoff condition holding sample size constant.

The Congruency between Global and Local Model Evaluation. The global model evalua- tion can provide three possible outcomes: trivial, severe, or inconclusive. The local model evalu- ation can provide four possible outcomes: trivial, severe, inconclusive, or underpowered. I use a contingency table to see the interaction between the results from both methods. The contingency table is used to see whether global or local fit evaluation is unnecessary to be used. One method is unnecessary if all information from the method is already provided by the other method. For example, global fit evaluation would be unnecessary if the following conditions are satisfied. (a) When the global evaluation provides trivial or severe outcomes, the local evaluation provides the

same outcomes. (b) When the global evaluation provides inconclusive outcomes, the local eval- uation provides trivial, severe, or inconclusive outcomes. In this case, the information from the global fit evaluation is covered by the information provided by the local fit evaluation; thus, it is not necessary to be used. If both methods are necessary, I will investigate the conditions under which global or local evaluations have a higher power to detect trivial or severe misspecifications. The contingency table is also used to examine the mismatches between both methods. I check the situation where one method indicates severe misspecification while the other indicates trivial misspecification. This situation should not occur.