ASSEMBLING A KNOWLEDGE-BASED ASSESSMENT

After a group of test items has been reviewed and edited, selected items are organized to form the knowledge test or questionnaire. For this to be done properly, the designer must decide the number of items to be included and the order in which the items are to be presented to the participants. Directions for the participants and a scoring key must be developed. Each of these steps must be taken carefully if the test items are to realize their maximum value.

Arranging Test Items

Test items should be organized on the basis of one or more of three characteristics: the subject matter, type of item and level of difficulty. Arranging test items according to subject matter means that the test items are grouped according to a set of subject matter topics. This arrangement appeals to participants since they see the test as a miniature of the materials to which they have been exposed. This is the most common arrangement and is recommended in most situations.

When the items are grouped according to type, all similar test items are placed together, including true-false, multiple-choice and matching items. This grouping simplifies the directions given to the participants. It is advisable for the designer to restrict the number of different types to as few as conveniently possible. When participants are confronted with a number of types of test items, they have a tendency to lose concentration and may not do well as a result.

Test items can be grouped according to their levels of difficulty if they happen to vary appreciably; that is, the easy ones first, the more difficult next and the most difficult last. Such an arrangement has advantages for average and below-average participants. With this kind of test they use the time allowed more efficiently, and morale is improved. When later they encounter the more difficult test items, they will have time to deal with them. Even if they fail to answer some, as will very likely happen, the resulting disappointment will be moderated by the knowledge that they already have answered others correctly.

The designer cannot expect to use all three ways of arranging test items simultaneously. In reality, all that designers can hope to do is find the best possible compromise among the three possibilities.

Giving Directions to Participants

To perform to the best of their abilities, participants must know the purpose of the test and must be thoroughly familiar with its mechanics. The designer formulates directions that the participants read or that are read to them before they respond to any of the test items.

In giving directions, the trainer should emphasize the mechanics, rather than the purposes, of the test. The participants need a complete understanding of the ground rules under which they will take the test. This means that they must be aware of the time allowed, the manner in which they are to select and record answers, and the scoring system to be used.

Directions for selecting answers must be written carefully. Notice that the sample instructions presented earlier assume that the participant knows

nothing about the test items. This of course is an extreme assumption and, for the most part, an unnecessary one. Even so, stating directions with too much detail is better than stating them with too little.

For assurance that these directions are understood, practice test items may be included. These may consist of typical items that are answered by the participants before they begin the test itself. The participants are told the correct answers to practice test items so that they can verify their understanding of the directions.

Reviewing the Assembled Test

After the test is assembled and the directions are written, it is a good policy to review each part critically. The designer should consider each item from the participants’ point of view. The grammar should be checked and the following questions asked:

! Are there question items for all the objectives covered by this test? Does the number of test items adequately reflect the amount of time spent per objective?

! Is the number of items included in the test in direct proportion to their importance in training?

! Does each item really measure the participants’ attainment of the objective? If not, how could it be revised to do so?

! Is each set of directions clear? Do the directions apply to every item in the group, or do some items require specific directions?

! Is there plenty of space to write the response? ! Are tricky, obvious or irrelevant questions avoided?

! Is each item separate and independent from the rest of the items? ! Are similar items grouped together?

! Is the test designed so that it is easy to score?

! Will participants be provided with meaningful feedback about their answers?

When possible, the test should be field-tested (pilot-tested) before it is used in regular training programs. After the test has been administered to one or two groups of participants, the designer needs to analyze and

improve it. The weaknesses that are revealed should be corrected and the test continually revised.