Writing Good Multiple Choice questions

Background/Context
Assessment (quiz questions, exams, practical’s, activities etc) should be directly aligned to the stated learning objectives. And the content of the training should also align directly (and support) the learning objectives. This helps ensure that the assessment is valid (that you are testing knowledge/skills/abilities based on what was taught).

Content covered in an instructional setting is not (necessarily) content learned, so the evaluation is a measure of (1) how well the instruction supports learning, and (2) how well students recall, apply, implement what was learned. Multiple choice, or multiple answer questions can be suited to assessing knowledge, comprehension, analysis, but on the other hand are not useful for assessing physical skill/ability or environment-level decision making that requires physical context.

The following provides a basic overview for writing multiple choice questions.

The stem (poses the question)
A researcher who is working alone spilled a small sample of radioactive material onto the floor. Which finding has the greatest implication for their safety?
The distractors (are the options that are not correct)

1. The researcher notes that the material was liquid and contaminated their lab coat
2. The researcher notes that the material is airborne and they may have breathed it in
3. The researcher notes that the material was solid and remained intact
4. The researcher notes that the material was enclosed

Example 1: Stem is meaningful:
What characteristic is consistent in mitochondrial genomes across species?

• Content (i.e, types of genes)
• Organization
• Size
• Lifecycle

Example 2: Stem is NOT meaningful:
Which of the following statements are true?

• mitochondrial genomes are relatively consistent in content (i.e, types of genes)
• mitochondrial genomes are relatively consistent in organization
• mitochondrial genomes are relatively consistent in size
• mitochondrial genomes are relatively consistent in lifecycle

• Is your stem meaningful?
  The stem should be meaningful by itself and should present a definite problem. A stem that presents a definite problem allows the learner to focus on the learning outcome
• Does your stem contain irrelevant material?
  If so, remove it. This can decrease the reliability and the validity of the test scores (Haldyna and Downing 1989).
  
• Is your stem negatively stated?
  If so, change it.  Learners often have difficulty understanding items with negative phrasing (Rodriguez 1997). In other words don’t use “Which is not correct.” If a significant learning outcome requires negative phrasing, such as identification of dangerous laboratory or clinical practices, the negative element should be emphasized with italics or capitalization.
  
• Is your stem in the form of a question?
  If so good! A question stem is preferable because it allows the student to focus on answering the question rather than holding the partial sentence in working memory and sequentially completing it with each alternative (Statman 1988). The cognitive load is increased when the stem is constructed with an initial or interior blank, so this construction should be avoided.

• Are all of the choices plausible?
  If not, improve them. Alternatives that are implausible don’t serve as functional distractors and should not be used. Using "funny" or distractors that are clearly improbable is poor practice. It is difficult to write meaningful questions and distractors so get help from an Instructional Designer to have a sounding board for iddeas and tips.
  
• Are choices stated clearly and concisely?
  They should be. Items that are excessively wordy assess students’ reading ability rather than their attainment of the learning objectives.
  
• Are there clues about which response is correct?
  Sophisticated test-takers are alert to inadvertent clues to the correct answer, such differences in grammar, length, formatting, and language choice in the alternatives. It’s therefore important that alternatives have the following:
• Grammar consistent with the stem.
• Structures that are parallel in form.
• Answer choices that are similar in length.
• Often the longest option is the correct option (and this becomes a gibe away).
  
• Did you use “all of the above” and/or “none of the above”?
  If so, reconsider. When “all of the above” is used as an answer choice, test-takers who can identify that more than one correct choice will know that the answer is “all of the above.” When “none of the above” is used, the test-taker simply needs to identify whether one of the choice answers is correct, and if so, they know if cannot be “none of the above.” They don’t have to think beyond that point. In either case, students can use partial knowledge to arrive at a correct answer.
• An alternative is to use (Select all that apply) with a multiple selection type question. This requires the learner to choose all of the choice answers that are correct.
  
• Does your quiz include “True or False” questions?
  If so, change these to multiple choice, or multiple correct. True or False questions simply measure a 50% chance of guessing correctly, and the correct answer is often easy to identify (often without having any understanding of the material).
• The DOE Training Handbook that EHS Training follows strongly discourages from using True/False
  

• Vanderbilt Center for Teaching Excellence, Writing Good Multiple Choice Test Questions
• http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
• Instructional Assessment Resources / Texas A&M
• https://www.utexas.edu/academic/ctl/assessment/iar/students/plan/method/exams-mchoice-write.php