The Art of Designing a
Multiple Choice Question
Dr. Amrit Krishna Mitra [W.B.E.S] Assistant Professor and Head
Department of Chemistry
Government General Degree College, Singur Hooghly, West Bengal, India
Executive Council Member [East Zone] Association of Chemistry Teachers
c/o Homi Bhabha Centre for Science Education, TIFR, Mumbai Email: [email protected]
Expected Learning Outcomes
1.
To exemplify the principles of designing good Multiple Choice Questions (MCQs) for assessing
Higher Order Thinking Skills.
Expected Learning Outcomes
2.
Identifying the drawbacks in MCQ setting.
3.
To prepare sample MCQs for testing memory, understanding,
application, analysis, evaluation and creativity.
A question is a word or words used to find out information.
Questioning is an important component of the teaching/learning process and is embedded
in quality instruction and strategic thinking.
Questions are used to teach as well as to assess student understanding and thus questioning
plays a critical role in the overall success of a classroom.
Anatomy of a Multiple Choice Question
The multiple-choice (MC) question is one of the most popular item formats used in educational
assessment.
A typical MC question has three parts:
A stem that presents a problem
The correct or best answer
Several distractors (i.e., the wrong or less appropriate options)
What is chiefly responsible for the increase in the average length of life in the USA during the last fifty years?
distractor— a. Compulsory health and physical education courses in public schools.
answer— *b. The reduced death rate among infants and young children
distractor— c. The safety movement, which has greatly reduced the number of deaths from accidents.
stem
alternatives distractor— d. The substitution of machines for human labor.
The stem may be in the form of either a question or an incomplete statement.
The purpose of the distractors is to appear as plausible solutions to the problem for those
students who have not achieved the objective being measured by the test item. Conversely, the
distractors must appear as implausible solutions for those students who have achieved the
objective. Only the answer should appear plausible to these students.
Multiple-choice test items are not a panacea
They have advantages and limitations just as any other type of test item.
Teachers need to be aware of these characteristics in order to use multiple-choice items
Advantages of Multiple-Choice Questions
Versatility
Analyse phenomena
Apply principles to new situations
Comprehend concepts and principles
Discriminate between fact and opinion
Interpret cause-and-effect relationships
Interpret charts and graphs
Judge the relevance of information
Make inferences from given data
Solve problems
Validity
In general, it takes much longer to respond to an essay test
question than it does to respond to a multiple-choice test
item.
The composing and recording of an essay answer is a slow
process.
A student is therefore able to answer many multiple-choice
items in the time it would take to answer a single essay
question.
This feature enables the teacher using multiple-choice
items to test a broader sample of course content in a given
amount of testing time. Consequently, the test scores will
likely be more representative of the students’ overall
achievement in the course.
Reliability
Well-written
multiple-choice
test
items
compare favourably with other test item types
on the issue of reliability.
They are less susceptible to guessing than are
true-false test items, and therefore capable of
producing more reliable scores.
Their scoring is more clear-cut than short
answer test item scoring because there are no
misspelled or partial answers to deal with.
Efficiency
Multiple-choice items are amenable to rapid
scoring, which is often done by scoring machines.
This expedites the reporting of test results to the
student so that any follow-up clarification of
instruction may be done before the course has
proceeded much further.
Essay questions, on the other hand, must be graded
manually, one at a time.
Limitations of Multiple-Choice Questions
Articulate explanations
Display thought processes
Furnish information
Organize personal thoughts
Perform a specific task
Produce original ideas
Provide examples
Number of 4-Alternative Multiple-Choice Items on Test
Chance of Scoring 70% or Higher by Blind Guessing Alone
2 1 out of 16 5 1 out of 64 10 1 out of 285 15 1 out of 8,670 20 1 out of 33,885 25 1 out of 942,651 Contd.
Difficulty of Construction
Good multiple-choice test items are generally more difficult and time-consuming to write than other
types of test items.
Coming up with plausible distractors requires a certain amount of skill. This skill, however, may be
Guidelines for Constructing Multiple Choice Questions
Guideline 1: The stem should be meaningful by itself and should present a definite problem
Guideline 2: The use of internal or beginning blanks in completion-type Multiple Choice questions should be avoided
Guideline 3: Use a negatively stated stem only when significant learning outcomes require it
Guideline 4: We should avoid irrelevant difficulty
Guideline 5: We should not attempt to increase the difficulty of an item by using unnecessarily complex or unfamiliar
vocabulary
Guideline 6: All distractors should be plausible
Guideline 7: We should avoid the use of complex MC format
Guideline 8: The relative length of the options should not provide a clue to the answer
Guideline 1:
The stem should be meaningful by itself and should present a definite problem
Example 1
Non-metals
A. cannot exist as solids at room temperature.
B. can combine only with metals to form stable compounds.
*C. usually have more than three electrons in the outermost shell of the atom. D. are usually found on the left hand side of the Periodic Table.
A common fault in MC questions writing is to have a brief, meaningless stem with problem definition revealed
in the options. In such cases, it can be difficult to see the intent of the item after reading the stem.
Our Feedback:
Students are faced with four true-false options; each is about non-metals but only option C is correct.
Furthermore, the four options cover a set of widely dissimilar chemical ideas so that evaluation by comparison is not possible. The stem can be judged to be clearly presenting a problem if it forces the options to be parallel in type of content.
Example 2
How many electrons could be found in the outermost shell of a non-metal atom?
A.
1
B.
2
C.
3
*D.
4
Our Feedback:
Example 2 demonstrates one way to make the stem become a definite problem.
Example 3
Which of the following statements concerning electrochemical cells is correct?
*A. There is a spontaneous chemical reaction in each electrochemical cell.
B. The e.m.f. of an electrochemical cell is measured in joules.
C. The anode is labelled (+) while the cathode is labelled (–).
D. The salt bridge provides electrons to complete the circuit.
Our Feedback:
Example 3 is a poorly written Multiple Choice Question.
The stem fails to present a definite problem and the four options appear to be a hodgepodge of chemical ideas.
Example 4
What is the main function of the salt bridge in an electrochemical cell?
*A. supply ions moving to the two half-cells
B. draw electrons from one half-cell to the other half-cell
C. keep the levels of solutions equal in the two half-cells
D. supply electrons to complete the circuit
Our Feedback:
Example 5
have the molecular formula C
nH
2n.
A. Alkanes
*B. Alkenes
C. Alkanols
D. Alkanoic acids
Guideline 2:
The use of internal or beginning blanks in completion-type Multiple Choice questions should be avoided.
Our Feedback:
Measurement specialists have advised not to use the completion format because a student has to retain the stem in short-term memory while completing the stem with each option.
If the completion format is unavoidable, the omission should occur toward the end of the stem rather than in the middle or at the beginning of the stem as shown in Example 5. Example 6 shows an improved version.
Example 6
Which type of organic substance has the molecular formula CnH2n?
A. alkanes *B. alkenes C. alkanols
Guideline 3:
Use a negatively stated stem only when significant learning outcomes require it
Most students have difficulty understanding the meaning of negatively phrased items. They often read through the negative terms such as not, no, and least, and forget to reverse the logic of the relation being tested.
Our Feedback:
Example 7 and Example 8 assess the same concept of chemistry, but some students may answer Example 7 incorrectly merely because of the word least.
Since least and concentrated are opposites, the phrase least concentrated is more difficult to understand than the phrase
most concentrated. Research has confirmed that the change from least concentrated to most concentrated will increase the
percent of correct responses.
Example 7
Which of the following solutions is the least concentrated?
A. 50 g of calcium carbonate in 100 cm3 of water B. 60 g of sodium chloride in 200 cm3 of water C. 65 g of potassium nitrate in 100 cm3 of water *D. 120 g of potassium sulphate in 200cm3 of water
Example 8
Which of the following solutions is the most concentrated?
A. 50 g of calcium carbonate in 100 cm3 of water B. 60 g of sodium chloride in 200 cm3 of water *C. 65 g of potassium nitrate in 100 cm3 of water
Our Feedback:
Often negatively phrased stems are useful if we want to assess whether students can identify dangerous laboratory practices that may damage expensive equipment or result in bodily injury, and which should not be carried out.
Example 9 is an example of such an item. However, when a negative term is used, it should be emphasized by being underlined or capitalized. Replacing the negative term with the word except can sometimes improve clarity, as illustrated in example 10. Few students would overlook the negative element in the stem because the word except is deliberately placed at the end of the stem and is capitalized.
Example 9
Water-type extinguisher is not suitable for putting out fire caused by burning *A. alcohol. B. cotton. C. paper. D. wood.
Example 10
Water-type extinguisher is suitable for putting out fire caused by burning all of the following EXCEPT
*A. alcohol. B. cotton. C. paper. D. wood.
Guideline 4:
We should avoid irrelevant difficulty
The difficulty of an item should not be increased by incorporating more complicated information in the stem than is necessary.
Example 11:
What volume of water should be added to 57.35 cm3 of 1.96
M NaCl in order to dilute it to 1.50 M? *A. 17.59 cm3
B. 42.65 cm3
C. 74.94 cm3
D. 112.41 cm3
Example 12:
What volume of water in cubic centimeters should be added to 60 cm3 of 2.0 M NaCl in order to dilute it to 1.5 M?
*A. 20
B. 40
C. 80
D. 120
Our Feedback:
If we want to assess whether students can solve dilution problems using the concept of molarity, Example 11 contains confounding detail. The values used in Example 12 will assess the same learning outcome and will avoid irrelevant sources of difficulty and error.
Guideline 5:
We should not attempt to increase the difficulty of an item by using unnecessarily complex
or unfamiliar vocabulary.
Example 13
Which ion below is responsible for the
colour of the gemstone called Topaz?
A.
B.
*C.
D.
Cr3+
Cu2+
Fe3+
Mn3+
Example 14
Which ion below is probably responsible for
the colour of yellow gemstones?
A.
B.
*C.
D.
Cr3+
Cu2+
Fe3+
Mn3+
Our Feedback:
We should not attempt to increase the difficulty of an item by using unnecessarily complex or unfamiliar vocabulary, such as the word Topaz in Example 13. This item aims at testing students’ understanding of the types of ions that give a yellow colour. But if students do not know that Topaz is yellow, they are lost. Example 14 is a better measure of the same learning objective.
The purpose of chemistry Multiple choice tests is to assess students’ knowledge, understanding and problem solving, not reading proficiency.
Guideline 6:
All distractors should be plausible.
Teachers are encouraged to use common misconceptions of chemistry students for distractors.
Example 15
Ordinary soft drinks like Coca-Cola have a pH about
A. 1 B. 2 *C. 3 D. 4
Example 16
Ordinary soft drinks like Coca-Cola have a pH about
*A. 3. B. 5 . C. 6. D. 8.
Our Feedback:
Teachers are encouraged to use common misconceptions of chemistry students for distractors.
The correct answer in both examples 15 and 16 is pH 3. Many students have the misconception that a solution is drinkable only if its pH is 7±2.
Guideline 7:
We should Avoid the use of complex Multiple Choice format
A complex Multiple Choice question consists of a list of potentially correct answers called primary responses and a list of combinations of the primary responses called secondary options. Students have to select one of the secondary options in answering the item, as shown in Example 17.
Example 17
Which of the following chemicals is/are contained in town gas? (1) hydrogen (2) sulphur dioxide (3) carbon monoxide (4) gaseous naphtha A. B. C. *D. (3) only (1) and (2) only (2)and (4) only (1), (3) and (4)
Example 18
What is the major constituent of the town gas in Hong Kong? A. carbon monoxide
B. gaseous naphtha *C. hydrogen
D. methane
Our Feedback:
If students know that the primary response “sulphur dioxide” is untrue, they tend to pick option D because sulphur dioxide does not appear in options A and D and more than one primary response are usually included in the correct answer to a complex Multiple Choice question.
Although the complex Multiple Choice format may make the items more difficult, research has revealed that it is less discriminating and reliable than the single-answer format. Example 18 shows an improved version.
Guideline 8:
The relative length of the options should not provide a clue to the answer
It is common to express the correct response more carefully and at greater length than the distractors. However, research has indicated that longer options tend to result in higher response rates.
Example 19
Why is zinc better than tin if we want to protect a piece of iron from rusting by electroplating?
A. Zinc is cheaper than tin. B. Tin is toxic.
C. Zinc can prevent iron from contacting with water and air.
*D. Zinc is more reactive than iron and thus rusting is prevented even when the metal plating is broken.
Example 20
Why is zinc better than tin if we want to protect a piece of iron from rusting by electroplating?
A. The cost of extraction of zinc from ores is lower than that of tin.
B. Tin is a toxic metal and causes incurable diseases.
C. Zinc can prevent iron from contacting with water and air. *D. Rusting is prevented even when the zinc layer is broken.
Our Feedback:
In Example 19, test- wise students will notice that option D is much longer than the other options even without a good understanding of the concept of sacrificial protection, they will guess that the correct answer is D because it stands out from the others. However, in Example 20, the correct answer is shortened and two distractors are rephrased to the desired length. Although expanding the distractors can increase their specificity and plausibility, teachers should not load them with irrelevant lengthiness or false technicality.
Guideline 9:
We should avoid using “none of the above” or “all of the above” as an option.
The use of none of the above and all of the above as options in MC items is tempting to many teachers because they appear to fit easily into many items. However, many measurement specialists do not recommend the use of the option none of the
above.
Example 21
Which of the following substances would relight a glowing splint?
A. carbon dioxide B. chlorine
C. nitrogen
*D. none of the above
Example 22
Which of the following substances would relight a glowing splint? A. carbon dioxide B. chlorine C. nitrogen *D. oxygen
Our Feedback:
The correct answer for Example 21 is option D. A student may explain this way: “The correct answer is none
of the above because, as everyone knows, hydrogen relights a glowing splint.”
Another student may be surprised to hear that explanation: “What! The correct answer is not hydrogen, but
sulphur dioxide.” It does not matter; neither gas is listed. Thus, the correct answer could be selected with
misinformation. This item may be modified to form Example 22.
Example 23
Which statement is true of most plastics? A. They have no reaction with acids. B. They can be moulded easily.
C. They are flammable. *D. All of the above.
Example 24
Which statement is true of most plastics? *A. no reaction with acids
B. difficult to be moulded C. nonflammable
D. good conductors of heat
Our Feedback:
The use of the option all of the above is also problematic. Example 23 is poorly constructed because a student may know that two of the three options offered are correct and this information can clue the student into selecting all of the above.
Thus, the option format allows students to select the correct answer on the basis of only partial rather than complete knowledge of the item. Example 24 shows an improved version.
Multiple
Choice
Questions
from Other
Example 25
California:
A. Contains the tallest mountain in the United States B. Has an eagle on its state flag.
C. Is the second largest state in terms of area. *D. Was the location of the Gold Rush of 1849.
Example 26
What is the main reason so many people moved to California in 1849? A. California land was fertile, plentiful, and inexpensive.
*B. Gold was discovered in central California C. The east was preparing for a civil war.
D. They wanted to establish religious settlements.
Our Feedback:
The stem is the foundation of the item. After reading the stem, the student should know exactly what the problem is and what he or she is expected to do to solve it. If the student has to infer what the problem is, the item will likely measure the student’s ability to draw inferences from vague descriptions rather than his or her achievement of a course objective. Hence, Example 26 is better than Example 25.
Example 27
If the pressure of a certain amount of gas is held constant, what will happen if its volume is
increased?
A. The temperature of the gas will decrease. *B. The temperature of the gas will increase.
C. The temperature of the gas will remain the same.
Example 28
If you increase the volume of a certain amount of gas while holding its pressure constant, its temperature will: A. Decrease.
*B. Increase.
C. Remain the same. Our Feedback:
Rather than repeating unnecessary words or phrases in each of the alternatives, place such material in the stem to decrease the reading burden and more clearly define the problem in the stem. Example 28 is better than Example 27.
Example 29
A word used to describe a noun is called an: *A. Adjective.
B. Conjunction. C. Pronoun. D. Verb.
Example 30
A word used to describe a noun is called: *A. An adjective.
B. A conjunction. C. A pronoun.
D. A verb. Our Feedback:
The word “an” in the stem of the Example 29 above serves as a clue to the correct answer, “adjective,” because the other alternatives begin with consonants. The problem has been corrected in the better example by placing the appropriate article, “an” or “a,” in each alternative. Undoubtedly, Example 30 is better than Example 29.
Example 31
When conducting library research in education, which of the following is the best source to use for identifying pertinent journal articles?
A. A Guide to Sources of Educational Information. *B. Current Index to Journals in Education.
C. Resources in Education.
D. The International Encyclopedia of Education.
Example 32
When conducting library research in education, which of the following is the best source to use for identifying pertinent journal articles?
A. A Guide to Sources of Educational Information. *B. Education Index.
C. Resources in Education.
D. The International Encyclopedia of Education.
Our Feedback:
When a word or phrase in the stem is also found in one of the alternatives, it tips the student off that the alternative is probably the answer. Example 32 is much better than Example 31.
Some useful hints to construct a good multiple choice question
Base each item on an educational or instructional objective of the course, not trivial information
Try to write items in which there is one and only one correct or clearly best answer
The phrase that introduces the item (stem) should clearly state the problem
Include from three to five options for each item
Test only a single idea in each item
Be sure wrong answer choices (distractors) are at least plausible
A question has to be thought out well enough to make it an ideal one
One has to invest a lot of time in it
Once a question is framed, the maker should go back to it after a few days time and attempt to
improve it
Incorporate common errors of students in distractors
The position of the correct answer should vary randomly from item to item
Avoid overlapping alternatives
The length of the response options should be about the same within each item (preferably short)
[1] Cassels, J. R. T., & Johnstone, A. H. (1984). The effect of language on student performance on multiple choice tests in
chemistry. Journal of Chemical Education, 61(7), 613-615.
[2] Chase, C. I. (1964). Relative length of option and response set in multiple choice items. Educational and Psychological
Measurement, 24(4), 861-866.
[3] Ebel, R. L. & Frisbie, D. A. (1991). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice Hall. [4] Haladyna, T. M. (1999). Developing and validating multiple-choice test items. Mahwah: Lawrence Erlbaum. [5] Osterlind, S. J. (1998). Constructing test items. Boston: Klumer Academic.
[6] Stiggins, R. J. (1997). Student-centered classroom assessment. Upper Saddle River, NJ: Prentice-Hall.
[7] Cheung, D. & Bucat, R. (2002). How can we construct good multiple-choice items? Paper presented at the Science and
Technology Education Conference, Hong Kong, June 20-21, 2002.
