REVISED PROCEDURES FOR DEVELOPING READING-INPUT MATERIALS AND READING-STORAGE TESTSa
Ronald P. Carverb
University of Missouri-Kansas City
Abstract. Reading-input and reading-storage are two new, completely objective techniques, useful for measuring certain aspects of reading. Reading-input is similar to the cloze technique. Reading-storage appears to provide a good measure of reading comprehension. Reading-input and reading-storage materials are produced from prose materials using algorithms. A revised algorithm is presented for preparing reading-input materials which is easier to implement than the earlier version. A new algorithm is also presented for preparing reading-storage tests. This new algorithm produces a test quite different from that of its forerunner, and one much easier to implement. These two new techniques are compared to the multiple-choice and cloze techniques and found to encompass most of their advantages while compensating for most of their disadvantages.
Standardized procedures exist for developing a variety of information- processing type measures for prose materials (Carver, 1971a). One of these standardized techniques, reading-input, is similar to the cloze technique and is designed to replace the cloze technique as an indicant of the readability or difficulty of prose materials. Reading-input might be considered "second gener- ation" cloze. The usefulness of the reading-input materials has been demonstrated in several ways. A standardized reading test has been developed using the reading-input procedures (Carver 1971c). Reading-input scores appear to be as valid as cloze scores for estimating readability or input accuracy (Carver, 1975c). Also, reading-input provides a useful technique for facilitating learning from prose materials (see Carver, 1975a). Although the reading-input procedures produce valid and useful materials, the algorithm originally presented for producing the materials is unnecessarily complicated in some respects. One purpose of this article is to provide a simplified algorithm for the development of reading-input materials.
A second purpose of this article is to provide the algorithm for developing the
aThe preparation of this paper was supported in part by the Office of Naval Research Personnel and Training Research Programs, Contract No. 0014-72-C-0240.
bReprints may be requested from Dr. Carver, School of Education, University of Missouri- Kansas City, 5100 Rockhill Rd., Kansas City, Missouri 64110.
revised version of the reading-storage test. The reading-storage test was designed to replace existing reading comprehension measures, e.g., multiple-choice questions. In a series of unpublished pilot studies, the initially suggested version of the reading-storage test (Carver, 1971b) was found to be relatively insensitive to significant differences in comprehension. Recent research indicates that the revised reading-storage test developed by a new algorithm is more valid than the cloze test (Carver, 1974b) and about as valid as the paraphrase test (Carver, 1975b) for measuring the primary effects of reading.
The third purpose of this article is to compare the reading-input and reading-storage techniques with two of the most popular techniques used in reading, i.e., multiple-choice questions and cloze.
REVISED PROCEDURES FOR DEVELOPING READING-INPUT MATERIALS Reading-input materials and reading-storage tests are developed from a passage of prose material, i.e. sentences meaningfully arranged in paragraphs. Fig. 1 contains a sample passage from which the sample reading-input test in Fig. 2 was developed. The revised steps for producing reading-input materials follow.
Figure 1: An Original Passage From Which Reading-Input and Reading Storage Tests May Be Developed
This is our Post Office. It is in our city. Many people work here. There is a Post Office in every city in our country.
And Post Offices in every country in the world.
A Post Office helper must be honest. He must be a good worker. A Post Office helper handles lots of mail. A Post Office helper handles lots of money.
The Post Office sends letters and packages, magazines, and newspapers all over the world. It sends small animals and plants, too. It sends money for us. It saves money for us. It puts money to work for us, too.
Step 1. Numerically label each word in order from 1 to Nt, where Nt equals the total number of words. (The first word in Fig. 1, This, is given the numerical label 1, and this labeling process is continued until the last word in the passage, too, is labeled 102.)
Step 2. Select the parameter value, Nr, which is the number of words in the repetitive unit (Nr < Nt). For example, if eight is selected to be the repetitive unit (Nr = 8), the words 1 to 8 are in the first repetitive unit and words 9 to 16 are in the second, etc. (In Figure 2, Nr = 5. The first repetitive unit consists of the first five words-77izs is our Post Office-and the third repetitive unit consists of the third consecutive set of five words-Many people work here There.)
Carver 157 Figure 2: A Sample of the Results of Applying the Reader-Input Procedures
Tliis is ° o c°u n t ry Post Office. It is ° £e r e our city.
Many people ° ^f e here. There is a ° | °s t Office in every city n £ our country. And Post n ^ °i c e s in every country in p. J* world.
A Post Office ° ^e l p e r must be honest. He g r~i Everv
be a good worker, pi A *>os' Office helper handles D a n d o f m a U- A P o s t D officf8 h e l p e r h a n d l e s l o t s o f
D and.
D money.
The Post Office sends r-, w°r e r and packages, magazines,
« * D n e p e r s
aU over t h e world- D Helper
sends SmaDanimals and p, Pj ,a n s' too. It sends money r-. °Ji- us. It saves money p, I™ s us. It puts money rj ° A work for us too.
Step 3. Select the parameter, Xj, which is the position of the item within the repetitive unit Nr ^.Xj !> 1). An item consists of the correct word, i.e., the word from the original passage, and one or more alternative words which are incorrect.
(In Fig. 2, Xj = 3. The first item consists of the correct word, our, and the alternative word, country.) Since the position of the item within the repetitive unit does not vary, the numerical labels for the series of items is the series Xj + kNr
where K = 0 , 1 , 2
Step 4. Select the parameter value, Na, which is the number of alternative words for each item (0 < Na). (In Fig. 2, Na = 1.)
Step 5. Select the parameter value, N , which is the number of words preceding and following each item that make up the candidate alternative word pool, i.e., the population of words from which the alternative words are chosen (Np < Nt) . (In Fig. 2, Np = 25. The 25 words preceding each item, X| - 25, and
the 25 words following each item, Xj + 25, comprised the population of words from which each alternative word was chosen for each item.)
Step 6. For the first item word, Xi; randomly select a candidate alternative word from the pool of eligible words designated in Step 5.
Step 7. Check the candidate word against a set of possible exclusion conditions. These exclusion conditions are referenced by the values E = 0, E = 1, E = 2, E = 3, and E = 4. Any one or any combination of the E values 1,2,3, or 4, may be selected. (In Fig. 2, E = 1, 2 , 3 , and 4.)
If E = 0, then no exclusions are made.
If E= 1, then compare the selected candidate word with the correct item word, and with any other previously selected alternative words, to see that the same word is not used more than once in any item.
If E = 2 , exclude any of the words represented by the values (Xj ± 1), (Xj ± 2), (Xi + 3) . . . (Xj ± Nb) where Nb is the number of words following the item or the number of words proceding the item that are excluded (Nb < Nt). (In Fig. 2, Nb = 4 was chosen so that the candidate pool of words for an item was not among those immediately adjacent to the item. For item 3, which consists of the two words, post and work, the four words preceding this item-owr city Many people—and the four words following-here There is a—were excluded as candidate alternative words.)
If E = 3, select the parameter value(s), X=, which is the position of the word or words in the repetitive unit which will be excluded. (In Fig. 2, Xj = 3. For example, in the first repetitive unit, Word No. 3, our, was excluded and in the third repetitive unit the excluded word was No. 12, work. These words are the correct words for each item. The correct item words, i.e., Xj = Xj, were excluded for the convenience of the manual selection process, described later.) Since the position of the deleted words within the repetitive unit do not vary, the numerical labels for the series of deleted words is the series Xj + KNr where Xj represents one or more discrete values such that Xj <!Nr.
If E = 4, exclude the candidate word when it is the same word as any of the words represented by the series (Xj ± 1), (X ± 2), (X + 3), . . . (Xj ± Nc), where Nc
represents a value such that 1 <. Nc < Nt. (In Figure 2, Nc = 4. For example, in the third repetitive unit an alternative word cannot be the same word as any one of the four words preceding the item-our city Many people— or the four words following the item-here There is a.) This exclusion condition precludes the selection of an alternative word which also appears in a set of words immediately adjacent to the item.
If the candidate word is excluded by any one of the exclusion conditions selected above, then go back to Step 6 and begin again to select an alternative word.
Carver 1 5 9
If the candidate word is not excluded by any of the above exclusion conditions, the candidate word becomes an alternative word for the item.
Step 8. Check to see if all of the Na alternatives have been chosen, then go back to Step 6 and repeat the same procedure again for choosing another alternative word. If Na alternatives have been chosen, then go to Step 9.
Step 9. Identify the numerical value of the succeeding test item. This value is Xj + Nr. The alternatives for item Xj + Nr are chosen in the same manner as Xj, i.e., let Xj + Nr be Xj and repeat Steps 6, 7 & 8. Continue with the entire series of test items—Xj, Xj + Nr, Xj + Nr + Nr, . . . —until Nt is exceeded, the termination of the procedure.
Step 10. For each item, randomize the presentation order of the set of words, i.e., the correct word and the incorrect alternative word or words, so that the correct word is not always in the same order position within the item. The alternative words are given the same initial letter case as the correct word, e.g., if the initial letter of the correct word is an upper case letter then the initial letter of all alternative words must be upper case.
It may be noted that the user of the procedures provides the values of the following parameters at the outset: Nt, Nr, Xj, Na, & Np. If E =£ 0, then values of one or more of Nb, Xj & Nc may also need to be chosen at the outset. Table 1 contains a list of the parameters, their symbols, and values for the example presented in Fig. 2. Given these parameters, the selection of the alternative words for each item follow the flow chart presented in Fig. 3, starting with Step 6 of the rules just presented.
The output, or result, of the above steps is a reading-input passage, similar to the example in Fig. 2. The output looks similar to the original passage except that alternative words are added to the correct item words. The task for the S is to select and designate the correct word for each item, and the results are scored using a standard correction for guessing formula, i.e., number right, minus number wrong divided by Na.
The preceding description of the procedures for developing reading-input tests was designed and presented in a manner that could be implemented with a properly programmed computer. Therefore, these procedures have two disadvan- tages. First, these objective, mechanical-type procedures sometimes result in ambiguous alternatives, e.g., sometimes the alternative wrong words fit the context as well as the correct word. Second, the procedures are complex, thus making it difficult to implement them.
The disadvantage of ambiguous choices can be overcome by editing, as was done in the development of the tests on the paragraphs in Reading Progress Scale (Carver, 1971c). The subjective judgment of the test editor as to whether each item
Table 1
Symbols and Examples Values for the Parameters
Parameter Input Parameters
- Total number of words in the body of prose
- Total number of words in repetitive unit
- Order position of test item in repetitive unit
Alternative Parameters
- Total number of incorrect alternatives per item
- Total number of words preceding and following an item which are included as possible candidate alternatives
Symbol
Nt Nr
* i
Na
N_
Parameter Values for Figure 2 Example
102 5 3
1
25 - Exclusion Possibilities E 1,2,3,4,
Total number of words preceding and following an item which are excluded
as possible candidate alternatives... Nb 4 Order position of words within the
the repetitive unit which are excluded
as candidate alternatives Xj 3 Total number of words preceding
and following an item which cannot include the same word as an
alternative N. 4
was ambiguous resulted, for example, in seven "vetoes" out of eighty items on Form 2 of this test. After an item was vetoed because of ambiguity, the alternative selection steps were executed again and another alternative word was chosen to replace the vetoed word. What is gained by eliminating item ambiguity must be weighted against the loss of objectivity in the process. Since the number of items vetoed seems to be low in proportion to the total number of items, and since the vetoed items are replaced using an objective procedure, the advantages gained by
Figure 3: A Flow Chart for Choosing Alternative Item Words in the Reading-Input Algorithm
Start the selection of alternative words for X,.
Step 6. Randomly select a candidate alternative word from among the pool of words contained within the limits
and X.. - K,
Test candidate word for possible exclusions.
Compare the number of alternatives chosen with N
Go to the next item word, Xj + Nr, let it be , and compare it with
t
editing out the ambiguous items may outweigh the disadvantages in certain situations.
The disadvantages of the complexity of the procedure are largely superficial.
By following certain manual developmental procedures, the alternative words for a 30-item reading-input passage can be produced for a 150-word prose passage in about 15 minutes. These manual steps are as follows:
Step 1. Select the parameter, Nr, the size of the repetitive unit. For most research purposes, Nr = 5, the same as regular cloze.
Step 2. Select parameter, Xj, the location of the item within each repetitive unit.
Step 3. On the prose itself, mark out the correct word for each item. For example, if Nr = 5 and Xj = 2, then the second word in the prose should be crossed out, and then every fifth word thereafter should be crossed out.
Step 4. Write or type the remaining words on a page in a column with the Nr - 1 words between items typed on each row. Leave space at the right hand margin of the page for the items (i.e., Na + 1 lines per item), and skip a line to designate a new paragraph. Figure 4 is an example of the worksheet produced by Steps 1-4.
Step 5. A random number table may be consulted to mark on the work sheet which of the word positions will contain the correct word. For example, odd numbers in a sequence of random numbers can be used to designate the upper position as containing the correct answer and even numbers can be used to designate the lower position as containing the correct answer.
Step 6. Copying from the original prose, write in the correct words for each item on the work sheet in the position designated in Step 5.
Step 7. Select the parameter, Np, which is number of words preceding and succeeding an item which are included as possible candidate alternatives. Select the exclusion conditions and the subsequent Nb, Nj & Nc parameters as needed. The manual implementation of the steps is greatly facilitated when: (a) Np is a multiple of Nr, (b) Nb = Nr - 1, (c) and Xj = X;. For example, consider the Fig. 2 example:
(a) Np = 25, and 25 is a multiple of 5, the Nr value, (b) Nb = 4, i.e., Nr - 1, so that the row of four words that the item is on is excluded as well as the following row of four words, (c) Xj = 3, i.e., Xj = Xj, so that all of the correct item words in the final column of items may be excluded.
Step 8. Assuming that the recommended parameter values in Step 7 are adopted, Step 8 involves three substeps. These substeps involve the use of a series of random numbers and they will be explained in concrete terms using the Fig. 2 parameters as examples.
Substep 8A. Enter a table of random numbers and select a one digit number.
Carver 163 Figure 4: A Sample Work Sheet for Manually Developing Reading-Input Materials
This is Post Office. It is our city. Many people here. There is a Office in every city our country. And post in every country in world.
A Post Office must be honest. He be a good worker.
Post Office helper handles of mail. A Post
helper handles lots of
The Post Office sends and packages, magazines, and all over the world.
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
—
If it is an odd numbered digit, the candidate word will be in the pool of words preceding the item. If the random number is an even numbered digit, the candidate word will be in the pool of words following the item.
Substep 8B. Inspect the subsequent one digit numbers in consecutive order until one of the digits 2, 3, 4 or 5 is found. This digit will designate the number of
rows following or preceding the item wherein the candidate alternative word will be found. Row number 1 will always be excluded when Nb = Nr - 1. The upper limit number, 5 in this case, will always equal Np / Nr.
Substep 8C. Continue to inspect the series of digits until one of the digits 1, 2, 3, or 4 is found. This digit will designate the order position of the candidate alternative word within the row. The above set of digits will always correspond to the number of words per row, i.e., 1 through Nr - 1.
To make Step 8 even more concrete, consider the following example: Given the series of random numbers—810241 1 6—for choosing the candidate word for item 9, in Fig. 2, it can be determined that the candidate alternative word is the fourth word in the second row following the item. This is exactly the position from which the candidate word, handles, was chosen in item 9 of Fig. 2.
Step 9. Check the candidate word for the item against the correct word (and any previously selected alternative word when Na > 1) to see if it is the same word as the correct word. This is the first exclusion condition, E= 1. If a match results, then the candidate word is rejected, and another candidate word is chosen according to Step 8.
Step 10. Select the parameter, Nc. The manual implementation of this step is facilitated when Nc = Nr- 1, i.e., when the number of words preceding and following an item which cannot include the same word as the alternative word is equal to the number of words in the same row and the following row.
Step 11. Assuming that the parameter value recommended in Step 10 is adopted, Step 11 involves a comparison between the candidate word and each word in the same row as the item and the row following the item. If the word matches one of these words, then it is rejected and another candidate word is chosen according to Step 8. If the word does not match any of these words then this word becomes an alternative word, and it is written in on the work sheet in the proper blank. The alternative word is capitalized when ever the correct word is capitalized, and when the correct word is not capitalized then the alternative word is not capitalized.
The remaining alternative words are chosen according to the same procedures, i.e., according to Steps 8 and 9 using the same parameter values as were used for the first alternative word.
PROCEDURES FOR DEVELOPING A REVISED READING-STORAGE TEST Reading-storage tests are also developed from a passage of prose material, i.e., sentences meaningfully arranged in paragraphs, such as Fig. 1. An example
Carver , 165 Figure 5: A Sample Reading-Storage Test
T IS
1. O POST O IT I IN H CITY. M PEOPLE 2. W HERE. O IS A POST O IN E CITY 3. I OUR C AND P OFFICES I EVERY C IN 4. T WORLD. A POST O HELPER M BE S HE 5. M BE A GOOD W A E OFFICE H HANDLES 6. S OF M A P OFFICE H HANDLES L OF 7. M THE C OFFICE S LETTERS A PACKAGES, M , AND 8. N ALL O THE W IT S SMALL W AND
9. H , TOO. I SENDS M FOR U IT S MONEY 10. F US. A PUTS M TO W FOR U , TOO.
reading-storage test for the prose in Fig. 1 is presented in Figure 5. The revised steps for developing reading storage tests follow.
Step 1. Select the word in the prose which will constitute the first word of the test. (Note: The first word in the body of prose does not necessarily have to constitute the first word on the test.) In Figure 5, the first word on the test is our.
Step 2. Starting with the first word, rearrange the prose into 10 words per line, each line being considered a test item.
Step 3. In each line of ten words, delete all letters except the initial letter of each of the 5 odd numbered words in the line and replace the deleted letters with a designating symbol such as a short underline. That is, every other word starting with the first should be represented by only its first letter and a symbol signifying the deleted letters of the word. One letter words, such as "A" and "I", must have the designating symbol added.
Step 4. For each line containing the five initial letters, delete one of the five according to a random selection process.
Step 5. Replace each letter in each line deleted in Step 4 by randomly selecting one of the initial letters in the remainder of the body of prose in accordance with the following conditions: (a) the population of letters be restricted
to the 5 lines preceding (25 letters) and 5 lines following (25 letters) the item line in question, and (b) the replaced letter must not be the same letter as the one deleted.
Step 6. Retype the test using capital letters only.
The above six steps produce a reading-storage test. The task for the test is to recognize and designate the one letter out of each set of five per line which incorrectly represents the original word. The test is scored using the standard correction for guessing—rights minus YA wrongs.
COMPARISON ANALYSIS
This section will contain the comparisons among the reading-input (RI), reading-storage (RS), multiple-choice (MC), and cloze (CLZ) techniques. The technique of writing multiple-choice questions is probably the most popular reading measurement method as demonstrated by the fact that practically every stand- ardized reading test involves multiple-choice questions. Currently, the most popular measurement method in reading research seems to be the cloze technique. Thus, it seemed fruitful to analyze and discuss the new measures, i.e., RI and RS, by comparing them to the most popular measures. The four techniques will be analyzed in the following paragraphs with respect to nine different criteria.
Can be scored objectively. Only so called "objective" techniques will be compared. It necessarily follows that all four techniques have this type of objectivity since this is generally what is meant by an objective test. However, there is a certain amount of subjectivity which usually creeps into the scoring of the CLZ items since spelling deviations are usually allowed and subjective judgments may provide for some minor amount of score variation from one scorer to the next. One of the primary factors contributing to the popularity of the MC approach is the complete objectivity with which the correct responses can be scored. Both the RI and RS tests incorporate this same advantage since the items are no different from MC items from a scoring standpoint.
Relationship between the amount of prose and the number of items. The MC procedure is the only one of the four approaches which does not automatically provide for the derivation of items which are systematically related to the size or quantity of the reading material. The CLZ approach calls for the deletion of every nth word. It has been suggested by Rothkopf (1968) that the CLZ technique has its historical roots in the completion procedure used by Ebbinghaus in 1897, and use of the CLZ technique was referred to by Rothkopf as the completion technique.
However, it seems desirable to keep the completion type of test separate from the CLZ type of test since the CLZ technique involves some type of unbiased algorithm
Carver 167 when constructing items. Thus, it seems appropriate to continue to give credit to Taylor (1953) instead of Ebbinghaus for developing a refinement of the completion type of test. The popularity of the CLZ technique is no doubt partly attributable to this systematic method for directly relating the units of measurement to the reading material. The RI and RS techniques both encompass this same type of advantage in that both derive items by an algorithm that directly relates the number of words in the passage to the number of items.
Objectivity in test development. The MC type of test has traditionally been termed an "objective test." However, the objectivity of the MC approach is connected with the objective scoring of the items, not with their objective development. MC items are traditionally developed subjectively, i.e., their develop- ment is generally recognized as an art (Davis, 1968). As for the CLZ technique, another important factor which contributes to its popularity is that it provides for the objective development of items. For example, an objective test may be developed on any prose material by simply deleting every fifth word or every fifth non-function word. The RI and RS items are also objectively developed according to the procedure outlined earlier. The RI test may involve a small amount of subjectivity if it is necessary that no ambiguous choices are provided.
Ease of item development. A major disadvantage of the MC technique is that a great deal of mental energy and time is required by an "expert" to develop good multiple-choice items. Undoubtedly, one of the major factors contributing to the popularity of the CLZ technique is its relative ease of construction. It is quite easy to develop CLZ items on an existing prose passage by deleting every fifth word, for example. The ease of construction of RI and RS items interacts with several factors.
These item construction procedures can be programmed for computer use.
However, when manually developed they are more difficult and time consuming than the CLZ technique, but probably no more time consuming than the development of MC items. Furthermore, non-professionals can be trained to apply the RI and RS techniques.
Task time limits and task difficulty. The time limits and difficulty an individual encounters when completing the tasks presented by these techniques would depend upon the difficulty level of the material, the ability level of the individual, and the length of the prose passage. For comparison purposes, let it be assumed that the material is college level difficulty, the individuals are college students, and that the amount of prose is 100 words. When there is an item every fifth word on both the RI and CLZ techniques, about twice as much time is required to complete the CLZ material as compared to the RI material. For example, if two minutes were allowed for the 20 RI items, then about four minutes should be allowed for the 20 CLZ items. The proportion of items marked correctly
will be around .90, on the average, for the RI as compared to around .35 on the CLZ. Whereas, there may be grumbling from some Ss about the CLZ tasks, a corresponding RI task will probably elicit little or no grumbling from the Ss. The time limit for completing the RS items should be greater than for RI items but less than for the CLZ items, e.g., about three minutes to complete the 10 items on 100 words, as compared to the two and four minute estimates given above. The RS task is a relatively difficult task. Several Ss are likely to grumble about the task. The difficulty of the task is evidenced by relatively low proportion of items that are answered correctly, e.g., about .40 for college students and college level material. It is impossible to compare the multiple-choice technique with the other three techniques on this criterion because there is no standard for determining how many questions should be written on a 100 word passage or how difficult the questions should be.
Investigating readability. The MC technique has been used to investigate readability (see Klare, 1963). Yet, easy questions can be written on difficult reading material and difficult questions can be written on easy reading material. Because of the subjective development of the MC questions and their alternative answers, the method has little usefulness for investigating or measuring readability. Conversely, the most popular and probably the most valid use of CLZ is to do what Taylor (1953) intended it to do, i.e., to measure or predict the relative ease with which a particular piece of prose could be read. A CLZ test on a passage can be given to a group which has never read the original, undeleted passage, and the degree to which the group successfully fills in the blanks is a relative indicator of the readability of the passage, compared to other passages. The RI approach incorporates the same advantages with respect to readability as the CLZ approach (Carver, 1975c). The RS test is not appropriate for investigating readability. Although there is nothing to prevent the RS test from being used to investigate readability, it probably would not be as sensitive to differences as the RI test. This is because a RI test and a CLZ test will reflect variance between passages when administered without prior reading of the original passage, whereas the RS test administered under these same conditions will tend to approach zero variance since the scores are likely to approach zero.
Measuring the primary effect of reading. The primary effect of reading may be called comprehension, understanding, amount learned, or information stored. As noted at the outset, the most prominent method for measuring comprehension has been the MC technique, and traditional prose learning research has also employed this technique. Yet, ordinarily the MC technique has not focused upon producing items which reflect gain during reading. The MC items have usually been developed from a psychometric standpoint rather than an edumetric standpoint (see Carver,
Carver 169 1974a). As a result, the items have not been passage-dependent (see Tuinman, 1974). Instead of measuring the primary effect of reading, the MC technique has usually produced items which can be answered correctly without ever reading the passages upon which the items are based (e.g., see Carver, 1972a). Recent research has indicated that paraphrase, MC items written upon each sentence in a passage and designed to reflect gains, often are extremely sensitive as a measure of the primary effect of reading but sometimes are extremely insensitive (Carver, 1975b).
The CLZ technique has been lauded as a valid method for measuring reading comprehension (Bickley, Ellington, and Bickley, 1970), and Rothkopf (1968) regards it as a simple, quantitative method for estimating what is learned from written discourse. Yet, the CLZ technique has been shown to be practically insensitive to reading when CLZ have been administered pre- and post-reading.
Coleman and Miller (1968) found inconsequential gains in CLZ scores as a result of having read passages. Carver (1973) also found the same type of result. While reading and auding rates climbed from 75 to 450 words per minute, understanding ratings of subjects changed from 0 to 80 percent, but CLZ scores varied only from about 15 to 35 percent. The RI technique could be expected to be no more sensitive to gain due to reading than CLZ. Neither the RI nor CLZ techniques seem to be appropriate for measuring the primary effect of reading. Recent research on the RS technique indicates that it is sensitive to the primary effect of reading. It was compared to an experimental version of the cloze technique, which was designed to be highly sensitive to gain, and found to be equally sensitive to gain (Carver, 1974b). It was also compared to the paraphrase MC items, noted earlier, and found to be more consistent but somewhat less sensitive, on the average (Carver, 1975b). There appears to be no available technique which is completely objective and yet more sensitive to the primary effect of reading than the RS technique.
Inducing prose learning. The use of questions as adjunct aids to learning from prose has received a great deal of attention (e.g., see Carver, 1972b or Frase, 1968), and some of this research has involved the MC technique. CLZ has also been used to induce learning from prose (e.g., see Louthan, 1965), but this use of CLZ has not been extensive. Recently, the RI technique has been used to induce greater prose learning (see Carver, 1975a) in situations where attention wanes. The RI technique has also been shown to stimulate more general prose learning while requiring less time to complete than the MC technique, as it has been used with training materials (Carver, 1974c). Although the RI technique has not been directly compared to the CLZ technique as a method for inducing prose learning, it would seem that it should be at least equally effective while requiring considerably less time. Thus, it appears that the RI technique may be more efficient in inducing prose learning than
either the MC and CLZ techniques. It should be noted that when the RI technique is applied to reading materials which are used to facilitate learning, as opposed to their use as tests or readability measures, these materials are called "programmed prose" (Carver, 1975a).
The RS technique is not appropriate for inducing prose learning, except indirectly when it is used as a criterion test to stimulate motivation.
Increasing reading skill. There are commercially available, reading improve- ment materials which require passages to be read and multiple-choice questions on the passages to be completed. The CLZ technique has been used in research to facilitate improvement in reading skill (e.g., see Kennedy and Weener, 1973;
Kingston and Weaver, 1970) with mixed results (see review by Jongsma, 1971).
There has been no research using the RI technique, i.e., programmed prose, to increase reading skill, but it would seem to have great potential in this regard. In a manner similar to CLZ, it could be objectively developed from any existing prose material. It should also result in approximately the same amount of learning per items completed as CLZ, but it would not require as much time to complete so it should result in more skill improvement per a fixed amount of time. The RI technique appears to deserve further research as a reading skillbuilder. The RS technique is not directly appropriate for increasing reading skill.
DISCUSSION AND CONCLUSIONS
The primary advantage of the reading-input and reading-storage techniques is that they are completely objective. This not only means that the results of their use can be generalized to other situations, i.e., experimenter bias can be completely eliminated, but it also means that they offer the potential of being automatically produced for practical situations. The multiple-choice technique, no matter what its advantages, still must be produced by learned and experienced individuals at a relatively high cost in time and resources. The reading-input and reading-storage techniques may be manually produced at present by relatively unskilled individuals, and in the future computers should be able to mass produce these materials at relatively low costs.
The reading-input technique seems to encompass almost all the advantages of cloze, yet requires less time for the individual to complete. It appears that anyone who is considering the use of cloze in a research task or in a practical application, should contrast the expected advantages and disadvantages of cloze with those expected from the use of reading-input. It seems likely that in most situations wherein it is highly appropriate to use cloze, reading-input would offer a better technique than cloze. As noted earlier, it seems appropriate to regard reading-input
Carver 171 as second generation cloze.
The reading-storage technique seems to offer many of the advantages of the multiple-choice technique while overcoming many of its disadvantages. The reading-storage test appears to reflect gain better than regular cloze (Carver, 1974b) and about as well as paraphrase questions (Carver, 1975b). As an objective measure of the primary effect of reading, it appears that the reading-storage technique deserves consideration.
REFERENCES
BICKLEY, A. C , ELLINGTON, BILLIE J., & BICKLEY, RACHEL T. The cloze procedure: A conspectus. Journal of Reading Behavior, 1970, 2, 232-249.
CARVER, RONALD P. Procedures for constructing a variety of information- processing measures appropriate for prose materials. Silver Spring, Md.:
Revrac Publications, 1971. (a)
CARVER, RONALD P. A computer model of reading and its implications for measurement and research. Reading Research Quarterly, 1971, 6, 449-471.
(b)
CARVER, RONALD P. Manual for the Reading Progress Scale. Silver Spring, Md.:
Revrac Publications, 1971. (c)
CARVER, RONALD P. Comparisons among normal readers, speed readers, and clairvoyant readers. F. P. Greene (Ed.) Twenty-first Yearbook of the National Reading Conference. Boone, North Carolina: National Reading Conference, Vol. 2, 1972. [Pp. 150-155] (a)
CARVER, RONALD P. A critical review of "mathemagenic" behaviors and the effect of questions upon the retention of prose materials. Journal of Reading Behavior, 1972, 4, 93-119. (b)
CARVER, RONALD P. Understanding, information-processing, and learning from prose materials. Journal of Educational Psychology, 1973, 64, 76-84.
CARVER, RONALD P. Two dimensions of tests: Psychometric and edumetric.
American Psychologist, 1974, 29, 512-518. (a)
CARVER, RONALD P. Measuring the primary effect of reading: Reading-storage technique, understanding judgments, and cloze. Journal of Reading Behavior, 1974, 6, 249-274. (b)
CARVER, RONALD P. Improving reading comprehension: Measuring readability.
Washington, D.C.: American Institutes for Research, Final Report R74-2, May 1974. (c)
CARVER, RONALD P. Manipulating attention during reading using programmed prose. Journal of Reading Behavior, 1975, in press, (a)
CARVER, RONALD P. Comparing the reading-storage test to the paraphrase test as measures of the primary effect of reading. Journal of Educational Psy- chology, 1975, in press, (b)
CARVER, RONALD P. Measuring prose difficulty: The RIDE Scale and the
Rauding Scale. Unpublished manuscript, 1975. (c)
COLEMAN, EDMUND G. & MILLER, GERALD R. A measure of information gained during prose learning. Reading Research Quarterly, 1968, 3, 369-386.
DAVIS, FREDERIC B. Research in comprehension in reading. Reading Research Quarterly, 1968, 3, 499-545.
FRASE, L. T. Questions as aids to reading: Some research and theory. American Educational Research Journal, 1968, 5, 319-332.
JONGSMA, EUGENE. The cloze procedure as a teaching technique. Newark, Delaware: ERIC/CRIER and the International Reading Association, 1971.
KENNEDY, DELORES L., & WEENER, PAUL. Visual and auditory training with the cloze procedure to improve reading and listening comprehension. Reading Research Quarterly, 1973, 8, 524-541.
KINGSTON, ALBERT J. & WEAVER, WENDELL W. Feasibility of cloze techniques for teaching and evaluating culturally disadvantaged beginning readers. The Journal of Social Psychology, 1970, 82, 205-214.
LOUTHAN, VINCENT. Some systematic grammatical deletions and their effects on reading comprehension. English Journal, 1965, 54, 295-299.
ROTHKOPF, ERNST Z. Textual constraint as a function of repeated inspection.
Journal of Educational Psychology, 1968, 59, 20-25.
TAYLOR, WILSON L. "Cloze Procedure": A new tool for measuring readability.
Journalism Quarterly, 1953, 30, 415-433.
TUINMAN, J. JAAP. Determining the passage dependency of comprehension questions in five major tests. Reading Research Quarterly, 1974, 9, 206-223.
