Analysis of the Innovative Domain

Following the same procedure as used in analysis of the Classical domain, we applied the IRT technique to the Innovative domain. This domain consisted of eight items: A (Non-traditional teachers), B (School choice) , E (Performance pay scale), G (District leadership), H (Year-round

schools), I (Charter schools), J (Data-driven decision making),and K (School leadership). Two

grm models were fitted to the data, with constrained and unconstrained slopes. The likelihood

ratio test indicated that the unconstrained model is a better option (p − value < 0.001). This

is clearly indicated in the slopes of the OCCs and ICCs in Figures 2.13 and 2.14. The curves

appear to have two distinct groups with very different slopes. The first one consists of the two items G and K, whose slopes are markedly steep, and the second consists of all the other six items (A, B, E, H, I, and J), whose slopes are fairly flat.

40

Figure 2.13 ICC curves. The middle categories in Items G and K have a narrow unimodal shape while for the other items the curves are very flat. For items E, B, H, and I the score of 1 seems to be the most likely response for a large group of people, especially for item I (Charter schools). On the contrary, scores of 4 or 5 are the most likely selected options, and 1 is a seldom used score.

Notice the visible black curves for items B, E, H, and I. For these items a score of 1 is the most probable score, particularly for item I (Charter schools). This is the most “difficult” item to endorse, and only those with a very high attitude score, at least in the upper range of θ > 2.8 , would acknowledge this item as somewhat important or more. This is in contrast to the pattern of item C, Professional development. Items B, E, and H, are also “difficult” because endorsing these items with a score of 4 or higher (very important to extremely important) would require an attitude larger than 2, essentially in the upper 2% of the respondents. The black curves for these items span dominantly across the attitude range, revealing the fact that for these items the score of 1, or Strongly Disagree, is the most likely score. That is, most respondents did not

support Charter School or Year-round school, School choice, and Merit-based performance pay as approaches to improving learning outcomes.

Figure 2.14 Plot of the OCC curves. Categories in Items G and K have similar and very steep slopes while the other six items have very flat slopes.

42

To assess the fit of the model, we again utilized the Kullback-Leibler discrepancy index. Results from 500 simulations reject the null-hypothesis (p − value = 0.018), indicating that the unconstrained model may have some deficiencies in describing the data.

We conducted a number of exploratory procedures to attempt to determine the manner in which the model does not adequately reflect the patterns in the observed data. In particular, we examined the number of response patterns and the proportion of responses in each category for each question. Visually, they did not appear to be major discrepancies between the actual data and the data simulated from the fitted model. Neither were we able to detect consistent patterns in discrepancies between actual and simulated data sets. This led us to conjecture that the small p-value from our goodness of fit procedure primarily reflects the effects of a large amount of data resulting in a test that is highly sensitive to small departure from the proposed model. To investigate this conjecture in more detail we conducted the entire analysis with a data set that resulted from randomly selecting half of each gender-stratified sample from the actual data. This reduced data set was used to estimate the model, examine inferential conclusions, and conduct the goodness of fit procedure. Results were similar to the full analysis in terms of substantive inference, but the goodness of fit test returned a p-value of 0.467, resulting in failure to reject the model as an adequate representation of the data. We interpret this outcome as support for our conjecture that the original p-value of 0.018 produced from the full data set was the result of a hyper-sensitive procedure as the result of a large sample size.

Table 2.7 shows the values of the estimated item parameters. In general, the category

estimates cover the whole spectrum of the attitude scale. The thresholds of items G and H spread to the negative side and are closer together; thus the ICC curves peak near the center of the b−parameters. These items can differentiate among three respondent groups of low, average, and high attitudes. For the other items, the categories are very far apart, which might add information about the latent trait at different locations. However, the ICCs were very flat, indicating that the items were unable to reliably distinguish low and high attitude groups.

Somewhat Moderately Very Imp Extremely Slope

A. Non Traditional Teachers -3.42 -0.74 2.36 5.70 0.43

B. School Choice -1.10 0.46 2.43 5.25 0.50 E. Performance pay -2.69 -0.86 1.06 3.41 0.50 G. District leadership -1.67 -1.02 -0.42 0.56 3.69 H. Year-round schools -0.60 1.11 2.86 5.09 0.55 I. Charter schools 1.32 2.83 4.14 6.04 0.66 J. Data-driven decision -6.56 -3.59 -1.90 0.27 0.75 K. School leadership -1.84 -1.29 -0.68 0.22 4.55

Table 2.7 Table of item parameter estimates, category thresholds and item discriminants of the Innovative domain.

The distribution of θ is centered around zero and spreads quite evenly across the range from

−2to 2, as seen in Figure2.15. Figure2.16presents the plots of the test information, standard

error, and the item information curves. The estimates of θ in the left plot appear to be more precise in the range from −2 to 1.5. The plot on the right shows that only leadership items G and K contribute the most to the estimates of θ. The curves of the other items are flat and thus are non-informative about θ. This finding suggests that there are five distinct groups of board members each having a different attitude level on school and leadership leadership issue.

44

Figure 2.15 Plot of the distribution of the estimated attitude scores. The curve appears to have a normal distribution with a center at zero and standard deviation of 1.

Figure 2.16 Left: plot of test information and standard error of measurement. The estimates of θ appear to be more precise in the range from −2 to 1.5. Right: plot of item information curves. Only items G and K contribute the most to the estimates of