PPMC

Posterior predictive model checks (PPMC) were performed for the IRTree Model, the MNRM, and the MPCM for 2 of the 24 combinations of survey length, response options, and sample size. PPMC method were used to investigate a cell with large mean IMSE differences across the three models and a cell with minimal mean IMSE differences across the three models. The PPMC methods explored the practical significance of the model differences in terms of item-level and person-level fit. Item-level fit and person-level fit were explored as the discrepancy measures relate specifically to extreme response tendency. The first simulation condition explored was 500 responses to 20 items with 6 response options generated under the MNRM. In the condition, the average IMSE was 0.77 for the MPCM estimation, 0.94 for the IRTree Model estimation, and 1.47 when the MNRM was used for analysis. When 500 responses to 10 items with 4 categories were simulated under the MPCM, the resulting mean IMSE values were 0.46, 0.59, and 0.57 for analysis completed with the MPCM, the IRTree Model, and the MNRM, respectively.

The item extreme response rate PPP-values and the individual extreme response rate PPP- values were computed for 1 replication in each of the two simulation conditions. Plots were generated to visually compare the resulting PPP-values. PPP-values near 0.5 provide evidence of adequate model fit. PPP-values greater than 0.95 or less than 0.05 are an indication of inadequate model fit.

Figure 30 displays the item extreme response rate PPP-values using parameter estimates from the IRTree Model, the MNRM, and the MPCM for 500 simulated responses generated under the MNRM to 20 items with 6 categories. Values near the horizontal reference line, PPP-value = 0.5, indicate model recovery of the item extreme response rate. Values above 0.95 indicate overestimation of the item extreme response rate and values below 0.05 indicate underestimation of the item extreme response rate. The results from the PPMC measure show that the MNRM did not capture the pattern of item extreme responses for most items. The PPP-values from the IRTree Model and the PPP-values from the MPCM all fall within the Bayesian credible interval (0.05 to 0.95). The PPP-values calculated under the MPCM fall closer to 0.5 than those calculated under the IRTree Model.

Figure 31 displays the individual extreme response rate PPP-values using parameter estimates from the IRTree Model, the MNRM, and the MPCM for 500 simulated responses generated under the MNRM to 20 items with 6 categories. Compared to the PPP-values calculated using the item extreme response rate discrepancy statistic, the PPP-values for the individual extreme response rate discrepancy statistic have more scatter for each of the three models. This may be a result of the low sample size. Observed and predicted frequencies of individuals at a few of the 21 unique extreme response rates are scarce. There were zero individuals predicted to have an extreme response rate of 0.10 or below in any of the 2,000 replicated data sets for the MPCM.

Additionally, there were no individuals predicted to get an extreme response rate of 0.05 or below in any of the 2,000 replicated datasets under the MNRM and the IRTree Model. The MNRM had inadequate fit across most response rates. The IRTree Model and the MPCM, however, had adequate fit across most individual extreme response rates. The pattern of PPP-values across the IRTree Model and the MPCM are indistinguishable in terms of overall person-level model fit.

For 500 responses simulated under the MNRM to 20 items with 6 categories, PPP-values from the item extreme response rate discrepancy statistic and PPP-values from the individual extreme response rate discrepancy statistic indicated that the MNRM had the worst model fit and the MPCM had the best fit. This result provides evidence that the difference in mean IMSE seen in Table 24 relate to meaningful differences of model fit.

Figure 30. Item extreme response rate PPP-values for 500 simulated responses to 20 items with 6 categories under the MNRM

Figure 31. Individual extreme response rate PPP-values for 500 simulated responses to 20 items with 6 categories under the MNRM

The second simulation condition considered was 500 responses simulated under the MPCM to 10 items with 4 categories. Figure 32 displays the PPP-values for the item extreme response rate discrepancy statistic computed under the IRTree Model, the MNRM, and the MPCM. The PPP-values associated with the MNRM and the PPP-values associated with the MPCM closely hover around 0.5. This result is an indication of adequate model fit with respect to the item extreme response rates across all items. The PPP-values produced under the IRTree Model contain more scatter across the items, however, they all are within the credible interval.

Figure 33 displays the PPP-values for the individual extreme response rate discrepancy statistic computed under the IRTree Model, the MNRM, and the MPCM. The PPP-values for the MNRM indicate inadequate person-level fit. Most PPP-values for the MPCM and the IRTree Model fall within the credible interval but have significant scatter. Three PPP-values fall outside the credible interval for the IRTree Model and two PPP-values lie outside of the credible interval

for the MPCM. Individual extreme response rates are susceptible to scare frequencies when only 500 responses are analyzed.

The results of the posterior predictive model checking performed using parameter estimates from the IRTree Model, the MNRM, and the MPCM for 500 responses simulated under the MPCM to 10 items with 4 categories were inconsistent. The MPCM and the MNRM were determined to have adequate item-level model fit. The IRTree Model was determined to have a lower degree of item-level fit than the other two models. The item-level model fit results match the pattern of the mean IMSE seen in Table 25. The mean IMSE was slightly larger for the IRTree Model than the relatively equal mean IMSE values determined using the MNRM and the MPCM.

The IRTree Model and the MPCM had the best person-level fit when the individual extreme response rate discrepancy statistic was analyzed. The IRTree Model and the MPCM have better quality person-level fit compared to the MNRM. However, this fit is not within the credible interval for all individual response rates. The pattern of person-level fit is inconsistent with the pattern of item-level fit and mean IMSE values. The pattern disagreement may be due to the low sample size considered.

Figure 32. Item extreme response rate PPP-values for 500 simulated responses to 10 items with 4 categories under the MPCM

Figure 33. Individual extreme response rate PPP-values for 500 simulated responses to 10 items with 4 categories under the MPCM