SEM techniques have been shown to provide a psychometrically rigorous avenue for assessing the construct validity of measurement instruments in the case of individual groups of participants. But often researchers are interested in comparisons between more than one participant groups. Researchers may want to determine if there are significant differences between Group A and Group B on a particular measure; or whether sex, culture, age or any other variable impacted the way the measures were completed. These substantive enquiries may be very applicable to the implicit attitudinal measures, for if such measures are deemed to be psychometrically adequate then it would be important to ascertain what information about implicit attitudes have been uncovered in a way that still accounts for error variance. Multiple-groups CFA may provide one means for achieving this. Usually, multiple- groups CFA is used to simultaneously test data from two separate participant groups. However, the methodology could also be applied to determine if there was a
significant difference between the latent means of the congruent and incongruent ‘groups’ of data, even if they are sourced from the same participant sample. Applied this way, multiple-groups CFA would provide a way of ascertaining whether an IAT effect had occurred for the sample whilst accounting for the confounding influence of error variance. However, before the average reaction times (or levels of a trait) for two different ‘groups’ can be compared, it is first crucial to assess whether a score of X by one group is equivalent to a score of X for the other group. If the trait scores are not comparable across groups then differences between groups in mean levels are potentially artifactual and may be substantively misleading (Reise, Widaman, & Pugh, 1993). This process of determining equivalency across groups is referred to as
testing measurement invariance, which aims to ensure that a particular measure is operating in the same way for different groups of people (Burns & Haynes, 2006). For the IAT, a test of measurement invariance would be required to ascertain whether an average reaction time of X for the congruent trials is equivalent to an average reaction time of X for the incongruent trials.
Multiple-groups CFA provides a strong analytic framework for evaluating invariance across distinct groups (Reise et al., 1993). By running simultaneous CFAs for two or more groups, the measurement and structural parameters of the comparative models are tested such that any group differences between the latent models are revealed (Brown, 2006). Multiple-groups CFA requires that constraints are applied to the models, such as like parameters, in a step by step process. This process enables a detailed examination of the equivalency of the measurement (measurement invariance) and structural (population heterogeneity) solutions (Brown, 2006). By applying this thorough approach to invariance testing it is possible to ascertain whether the factor structure, the structural equivalence, the intercepts, the error variance, the variance (or standard deviation) and the latent mean scores are the same for each group (Burns, Gomez, Hafetz, & Walsh, 2006). These tests provide a direct contrast of the aforementioned aspects of the groups in order to determine
comparability, or indeed, significant differences between the groups. Multiple-groups CFA require two separate input matrices and the analyses follow a specific
procedural format.
To begin the assessment of measurement invariance, the factor structure of Group A and Group B’s models are examined and compared. For the IAT, Group A could
refer to the congruent trial data, and Group B the incongruent trial data. The initial assessment of factor structure for these ‘groups’ is referred to as equal form invariance (or configural invariance) and it means the number of factors and the pattern of indicator-factor loadings is identical across groups (Brown, 2006). This is the least restricted model which subsequent models are evaluated against using the nested chi-square (χ2) difference test, a measure of the relative difference in chi- square value for two nested models. After establishing equal form (configural) invariance, the next series of analyses entail increasingly restrictive constraints. Factor loading equality is initially examined (metric invariance), then equality of intercepts (scalar invariance or strong factorial invariance) and finally the equality of the error variances (residual invariance or strict factorial invariance) (Brown, 2006). These four separate analyses comprise the assessment of measurement invariance.
Once measurement invariance is established, multiple-groups CFA can be applied to test the equality of latent means. This is the key aspect of multiple-groups CFA of interest for the IAT, as the analysis could reveal whether an IAT effect was present or not by showing whether there is a significant discrepancy in the latent means for the congruent and incongruent data. If the latent means for the congruent trials were found to be significantly smaller than the latent mean for the incongruent trials it would imply a positive IAT effect had occurred, that the expected attitudinal bias was revealed. The comparison of latent means is somewhat analogous to the comparison of observed group means completed using a t-test or ANOVA (Brown, 2006). However, unlike these traditional ways of determining whether an IAT effect has occurred, the comparison of latent means accounts for error variance, thereby providing a more accurate indication of the implicit biases evidenced.
Group comparisons of latent means are only meaningful if the factor loadings (metric) and indicator intercepts (scalar) have been shown to be invariant. As such, the comparison of latent means cannot occur until after the analyses of measurement invariance are completed. In order to compare two latent means, the latent mean of one model (the congruent trial data) is constrained to be zero, whereas the second latent mean (for the incongruent trial data) is allowed to be freely estimated. If this produces a significant latent mean for the second ‘incongruent’ group it indicates there is a significant difference between the means of the two groups, i.e. that an IAT effect was present for the sample. Figure 4.5 provides a conceptual model of this assessment of latent means using multiple-groups CFA. For a two group comparison, the difference between latent group means is equal to the latent mean for the second group and the sign of the second group’s latent mean provides a guide as to which mean was higher (Thompson & Green, 2006). For the example IAT data in Figure 4.5, a significant and positive result indicates the average reaction time for the incongruent trials was significantly slower than the congruent trials, demonstrating the expected IAT effect.
Figure 4.5. Multiple-groups CFA model assessing the IAT effect for Construct 1.
As outlined above, Multiple-groups CFA could provide one avenue to compare the congruent and incongruent IAT results to determine if an IAT effect had occurred. This approach would avoid the confounding influence of error variance on the results, potentially delivering a more accurate impression of the implicit biases of the sample. A similar approach would also be appropriate for the APT data.
Application 7: Testing for the Effects of Covariates on the Latent Factor