Smoothing approaches for empirical PRFs attempt to approximate the unknown response function from empirical response data. Hanning smoothing (Tukey, 1977; Velleman & Hoaglin, 1981) has been applied to PRF fit analysis in the Rasch context (e.g., Walker et al., 2018). In the present study, the Hanning function described in
Equation (3) was adapted to construct empirical GRFs. First, p-values were computed for each item based on subgroup responses. Second, the p-values were ordered in terms of latent item difficulty. Third, the Hanning algorithm was applied to the ordered p-values.
A smoothed estimate for item j (j =1, ..., J) was given by
ℎ' = (Y'K;+ 2Y' + Y'M;)/4 , (4)
where Y' is the observed item p-value computed from subgroup responses to item Z. The first and last p-values (Y; and YP) were left unsmoothed, following Walker et al. (2018). The smoothed p-value, ℎ', replaces observed p-values YQ through YPK; . The smoothing algorithm was then repeated on the smoothed p-values in an iterative process.
Appropriate number of iterations is the primary topic of Study 1.
Any smoothing approach involves balancing two types of errors: sampling variance and bias (Ramsay, 1991). In the present context, large sampling variance may result in over-flagging GRF patterns as aberrant (inflated Type I error rate). Large bias, however, may result in “smoothing over” significant deviations from expected patterns (under-flagging). As previously noted, in kernel smoothing approaches (e.g., Emons et al., 2004), the balance between bias and sampling variance is controlled by the user-
specified bandwidth. In Hanning smoothing, the balance can be controlled by
manipulating the number of smoothing iterations. Too many iterations may induce bias. Too few may inflate Type I error rate. The recommendation for generating empirical PRFs using Hanning is to set number of iterations equal to the person number-correct score (Engelhard, 2015; Walker et al., 2016; Walker et al., 2018). However, no studies have been found that have systematically investigated the relationship between Hanning iterations and accuracy. Also, for a GRF application, the appropriate number of iterations may be moderated by number of examinees and number of items.
Using simulated data, Study 1 investigated the accuracy of the Hanning procedure for generating empirical GRFs across item subset size, subgroup size, and number of smoothing iterations. Repeated samples of model-fitting (non-aberrant) item response vectors were simulated according to the Rasch model. Person and item parameters used to generate response probabilities were consistent with estimates derived from real test data from a K-12 educational testing context. The observed item difficulty estimates were used (Mean = 0.00; SD = 0.846; Min = -1.54; Max = 2.25). Person parameters (q) were drawn from N(1,1), which was consistent with the observed distribution of person estimates. The study explored 250 conditions obtained by fully crossing three independent variables:
1. Item subset length (2 levels). Item subset length had two assigned levels: 12 and 24. The real data test form contains 48 scored items, therefore accuracy for 24 items was compared to accuracy for 12 items. A 24-item subset represents splitting the test into two equal-length subsets, which could represent, for
example, the first and last half of a fixed form exam. A 12-item subset is closer to subset lengths associated with passage-based exams. In the real data test form, each item was associated with one of six passages, with six to nine items per passage.
2. Subgroup size (5 levels). This study investigated subgroups of 25, 50, 100, 500, and 1000. The variation in size reflects the fact that while test data for statewide K-12 testing is typically relatively large, some subgroups (e.g., ethnic minorities, English learners) may be relatively small. Recall also that the final subgroup size used in the proposed GRF approach includes only aberrant examinees. The final subgroup size may therefore depend on several factors: (1) the size of the total subgroup in the sample; (2) the person fit statistic used; (3) the flagging criteria used; (4) the incidence of aberrance in the subgroup population.
3. Number of smoothing iterations (25 levels). One to 24 iterations were explored. For comparison, an unsmoothed p-value condition (zero iterations) was also investigated. The zero-iteration condition is worth exploring because for large subgroup sizes, p-value may be sufficient to estimate an accurate GRF.
Each condition was replicated 100 times. The number of smoothing iterations is used to answer the research question for Study 1, with factor 1 answering research question 1a and factor 2 answering research question 1b.
Four outcome variables were computed on each of the 250 conditions. First, to investigate sampling variance, the mean (over replications) sum of squared errors was computed. That is, for each condition,
[[\] = 1 ^8 8_7`a$,'- − 7$,'-b Q , P ':; c a:; (5)
where 7`a$,'- is the empirical GRF for replication d (d = 1, … , ^) at item Z with Rasch scale location ,'. Total squared deviation (TSD) was computed as the squared difference between mean empirical GRF (over replications) and theoretical GRF, summed over items. That is,
f[g = 8 h7`a$,'-]− 7$,'-iQ P
':;
. (6)
Emons et al. (2004) suggest Equations (5) and (6) for investigating empirical PRF sampling variance and “bias”, respectively. Note, however, that in Emons et al.’s
formulation, SSEM contains both sampling variance and bias, as it involves differences between true (known) parameters and the estimates of those parameters. In addition, a more meaningful measure than TSD is mean absolute deviation (MAD), which puts differences between empirical and theoretical GRFs on the probability scale. For example, a MAD value of 0.2 would indicate the average deviation from the theoretical curve (over all items) is 0.2 probability. The calculation for MAD is
jkg =1 l8 mh7`a$,'-]− 7$,'-i Q P ':; . (7)
Results of Study 1 were used to guide Study 2 and Study 3 in terms of selecting a number of smoothing iterations that reflects the best compromise between sampling error and bias. To visualize the appropriate number of iterations, for each condition, Root Mean Square Error (RMSE) over j items for each replication (v) was computed and plotted against number of smoothing iterations. Like SSE, RMSE includes both sampling variation and bias. Like MAD, RMSE is on the probability metric. RMSE expresses the average error in estimating the GRF. Mean RMSE over all replications was computed as
nj[\] = 1 ^8 o 1 l8_7`a$,'- − 7$,'-b Q P ':; c a:; . (8)
The “optimal” number of iterations for a particular condition was defined asthe iteration number yielding the minimum RMSEM (or where further iterations yield no further decrease in RMSEM). Although the optimal number was defined in terms of RMSEM, all four measures discussed above were used to interpret results.