Sequential Probability Ratio Test (SPRT)

The sequential probability ratio test (SPRT) uses item-bank level probabilities and examinee responses to randomly drawn items from the item-bank to make classification decisions about an examinee. The following outlines the rule base and equations behind the SPRT approach to making classification decisions. Given a bank of items that assess mastery of a single learning objective, item-bank level probabilities can be either established

empirically or set by the test administrator to form the SPRT rule base presented in Table 5. SPRT is based on the Classical Test Theory psychometric model and uses an item-bank where item-bank level probabilities have been established. In the absence of specific information about the probability that a given examinee belongs to a specific classification group equal prior probabilities are assumed.

Table 5. SPRT Rule Base (adapted from Welch, 1997, p. 39)

Rule Description Conditional Probability

1A If the examinee is a master (M), the probability (P) of selecting an item that will be answered correctly (C) is .85

P(C|M) = .85

1B If the examinee is a master (M), the probability (P) of selecting an item that will be answered incorrectly (¬C) is 1 - .85 or .15

P(¬C|M) = .15

2A If the examinee is a nonmaster (N), the probability (P) of selecting an item that will be answered correctly (C) is .40

P(C|N) = .40

2B If the examinee is a nonmaster (N), the probability (P) of selecting an item that will be answered incorrectly (¬C) is 1 - .40 or .60

P(¬C|N) = .60

During testing using the SPRT approach, an examinee is randomly administered an item from the item-bank, their response is evaluated, and the SPRT rule base is used to evaluate the probability that they are a master or a nonmaster. The test continues until either the probability ratio of the examinee being a master versus a nonmaster goes above or below specific threshold values or the maximum number of items has been administered to the examinee.

The probability that the examinee is a master is the product of the prior probability of mastery, the probability that a master would get the specific number of items correct, and the probability that a master would get the specific number of items incorrect (the numerator in equation 10). The probability that the examinee is a nonmaster is the product of the prior probability of nonmastery, the probability that a nonmaster would get the specific number of

items correct, and the probability that a master would get the specific number of items incorrect (the denominator in equation 10). The probability ratio that the examinee is a master versus a nonmaster can be calculated via equation 10.

The probability ratio is then compared to two threshold values (see equation 11) that are based on the acceptable error rates for making either a false mastery decision (αFM) or a false nonmastery decision (βFN) set by test administers. If the probability ratio is between the two thresholds then no-decision can be made and testing must continue. If the probability ratio is less than both thresholds then a nonmastery decision can be made and if it is greater than both thresholds then a mastery decision can be made.

= |_{| #}# _{¬ |}¬ | _#¬#¬ (10) ! 1 − "! < < 1 − _! "! (11) Where: PR = Probability ratio

P(C|M) = Probability of a correct response given mastery

P(¬C|M) = Probability of an incorrect response given mastery

P(C|N) = Probability of a correct response given nonmastery

P(¬C|N) = Probability of an incorrect response given nonmastery

PM = Prior probability of mastery

PN = Prior probability of nonmastery

#C = Number correct responses from examinee with unknown mastery status

#¬C = Number incorrect responses from examinee with unknown mastery status

αFM = Error rate established a priori for making false master decisions βFN = Error rate established a priori for making false nonmaster decisions

3.2.2 An Example

The following example of a single test administration uses the SPRT rule base presented in Table 5 and equal probabilities of a false mastery decision and false mastery decision (αFM _{= β}FN = .025) so that there is a 5% chance that SPRT will make an incorrect classification decision. Equation 11 is used to establish the threshold below which the probability ratio value would lead to a nonmastery decision (PR < 0.026) or above which the probability ratio would lead to a mastery decision (PR > 39).

Table 6. Example of SPRT

R

Probability of R From: Probability

Examinee Is A: PR Test Decision Master Nonmaster Master Nonmaster

.5 .5 1 1 C .85 .40 .680 .320 2.125 Continue 2 ¬C .15 .60 .347 .653 0.531 Continue 3 C .85 .40 .530 .470 1.129 Continue 4 C .85 .40 .706 .294 2.399 Continue 5 C .85 .40 .836 .164 5.098 Continue 6 C .85 .40 .915 .085 10.833 Continue 7 C .85 .40 .958 .042 23.019 Continue 8 C .85 .40 .980 .020 48.916 Master

Table 6 summarizes eight repetitions of the procedure of administering a random item, evaluating the examinee response, and determining if mastery or nonmastery decision can be made. Each repetition and row in Table 6 corresponds to the administration of a new item. For every item administered, Table 6 details the probability of the response (correct or incorrect) from a master and a nonmaster from the SPRT rule base established during calibration, the subsequent probability that the examinee is a master or nonmaster, the probability ratio (PR) from equation 10, and associated test decision based on comparing the PR to the upper threshold (> 39) and lower threshold (< .026) detailed in equation 11.

After administering eight questions the examinee has answered all but one item correctly and enough data has been collected to enable a mastery decision to be made so the test may terminate. Note that before any items have been administered the probability that

the examinee is a master is the same as the probability that the examinee is a nonmaster and that the probability ratio is 1.

After the first item is administered and the examinee responds correctly equation 10 is used to calculate the probability ratio with the number of correct responses equaling one and the number of incorrect responses equaling zero. The probably ratio after one item indicates that the examinee is over twice as likely to be a master versus a nonmaster but testing must continue since the probability ratio is less than the upper threshold. When the examinee responds incorrectly to the next question the probability ratio is updated and now the examinee is more likely to be a nonmaster than a master. Again, the probability ratio is still between the two threshold values so testing must continue. From this point on the examinee answers all the items administered correctly and, consequently, the probability ratio steady climbs until it passes the upper threshold and a classification decision of master can be made.

3.3 Expert Systems Enhanced SPRT with Random Item Selection (EXSPRT-R)