How to Check the Quality of AI-Generated Incorrect Answer Options

AI quickly adds several other answers to a correct solution. The test looks complete: a question, list, and key. Yet one option may be an obvious joke, another accidentally correct, and a third different only because of an unfamiliar term. Such a set reveals little about whether students understand the topic. An incorrect option, or distractor, must be substantively connected to the task and distinguish subject knowledge from superficial recognition of the author's style, sentence length, or the most familiar wording.

Check distractors within the complete task. An answer that is wrong under one condition may be acceptable under another. Below is an editorial process for questions with one best answer. It does not guarantee psychometric properties without student performance data. All tasks and results in the examples are hypothetical. AI creates candidates and supports critical review; instructors confirm subject content, while a pilot helps check how actual participants in a particular course read the options.

Establish the Basis of the Correct Choice

Begin with an independently verified solution and explanation of why it meets the task conditions. Do not generate incorrect options before defining the objective and framework. If the key is uncertain, additional answers only multiply the problem. Record required assumptions, units, and detail level. For a best-next-step question, set the preference criterion. Distractors must be inferior to the key under this criterion, rather than because of a hidden authorial intention students cannot infer from the supplied material.

Next, identify the knowledge or reasoning error each alternative should reveal. This is not a diagnosis of a particular student. Selecting an option provides a signal for further consideration, not full access to a person's thoughts. The same choice may arise from misunderstanding, careless reading, or an accidental click. Use the connection with a typical error for design and explanation, rather than turning one question's result into a claim about stable abilities or a participant's learning profile.

Collect Plausible Errors from Subject Practice

Sources can include anonymized previous work, instructor observations, and analysis of solution steps. Distinguish an actually observed error from an author's hypothesis. If sources are unavailable, AI can suggest possible mistakes, but they must not be presented as confirmed student behavior. Ask which faulty step leads to each answer. An option without a reproducible connection to the task needs especially careful consideration or replacement before the next stage.

Finding good distractors is a separate research task. For example, Bitew and colleagues studied reusing candidates with contextual selection and instructor assessment. This examines a specific approach, rather than confirming the quality of any modern generation. The requirement to check candidates in context is useful in your work. A system suggesting a word from a similar topic does not mean it will become a meaningful alternative in a particular test item.

Give AI a Task About a Faulty Step

Instead of 'add three wrong answers,' supply the learning objective, verified solution, and error types you want to consider. Ask each candidate to include a faulty step, its result, and why it fails under the task conditions. Do not permit silent changes to numbers or wording. You need material for expert selection rather than an automatic final test. This format lets you reject candidates that sound convincing but arise from no understandable attempted solution to the actual task.

After receiving a list, check one option at a time. A model may correctly explain a general error yet miscalculate its consequence. Do not accept a description as proof of a number. For a conceptual question, check whether the explanation concerns the supplied conditions. If a candidate needs an additional hidden assumption, state this explicitly. Sometimes such an assumption makes an option useful; other times it creates a second acceptable key or an unrelated task within the main question.

Work Through a Numerical Example Without Random Values

A hypothetical task: five equally weighted assignments have scores of 6, 7, 8, 9, and 10; find the arithmetic mean. The sum is 40, and dividing by five gives 8. Candidate 40 may represent stopping at the sum without finishing the operation. Candidate 10 may result from dividing the sum by four. However, these are only the author's hypotheses about possible steps. They do not prove that students in your group actually make these particular errors.

A randomly added number may also be wrong, but does not explain what its selection tests. Compare it with a candidate derived from a specific faulty step. Then check the whole set: duplicate answers, an unambiguously specified mean, and any unagreed weights. If scores change, recalculate every option. Old distractors do not automatically remain meaningful under new conditions, even when the correct number happens to stay unchanged through symmetry or another feature of the example.

Find a Hidden Second Key

For every incorrect option, ask whether a prepared student could justify it without violating the task conditions. Ask AI to construct such an argument, then check it yourself. If it needs a different context absent from the task, that does not yet make the answer correct. If the conditions genuinely permit two interpretations, revise the task. Do not shift responsibility to students by saying they 'should have understood what the author wanted.' A test must assess a defined subject action from available data.

In a best-answer format, some alternatives may be generally relevant but meet the explicit criterion of the particular situation less well. NBME explains one-dimensional comparison of options for its subject-specific items. In a general course, preserve a visible basis for selection. If you cannot briefly explain why the key is better than every alternative under the same framework, the question needs revision. Do not label an acceptable professional alternative an 'error' merely because it was not first in the author's intention.

Check the Set's Homogeneity

Options should answer the same question at comparable levels of description. If one is a cause, another a tool, and a third a consequence, participants may find the key from form without the required knowledge. Similar length does not mean identical character counts. Comparable detail and structure matter more. Do not add a long educational explanation to the correct option while leaving others two words each. Put explanations in separate feedback after the task.

The University of Waterloo describes plausibility and similarity of alternatives as important requirements for this question format. During review, compare grammar, terminology, units, and degree of certainty. If only the key grammatically completes the stem, that is a technical cue. If every wrong answer contains obvious absolutes, participants may select a more cautious formulation without subject reasoning. Correct the cue itself rather than automatically concluding that every longer or more categorical sentence is always unsuitable for a test.

Distinguish Plausibility from a Deliberate Trap

A good distractor may reflect understandable incomplete knowledge. This does not mean hiding a condition in small print or building a question around a rare ambiguous word. Define exactly which skill the choice reveals. If an option works only through an inconspicuous wording change, it may measure attention to a trap instead of the target outcome. Such detail can sometimes suit precise document-reading tasks, but must be included in the learning objective with appropriate participant preparation.

Do not make an incorrect option ridiculous or humiliating. A joke can be rejected without knowledge, while associating it with a group can create an unnecessary stereotype. For a professional topic, showing an actual boundary of a concept's application is more useful. An incorrect method, for example, may be correct for another data type. Feedback then explains when it is appropriate. Students expand their understanding instead of merely memorizing forbidden answers without connections to the task's context and purpose.

Write an Explanation for Every Option

Feedback should return to the reasoning step: what was omitted or applied outside the conditions. Repeating 'this option is incorrect' is insufficient. For a numerical task, show the relevant operation; for a conceptual one, the distinction between objects or conditions. Avoid explanations introducing a new error or an unconditional rule where qualification is needed. Ask a subject editor to check feedback separately from the key. A correct choice does not guarantee correct reasoning in what students read after selecting an answer.

If no useful explanation can be written for an option, reconsider its role. It may be unrelated to the objective or duplicate another alternative. Do not increase the answer count for layout symmetry. A smaller verified set is better than additional random words creating an appearance of difficulty. Choose the number according to the course format and content. Do not transfer a recommendation about quantity from one context as a universal guarantee of the entire test's reliability, difficulty, or learning value.

Check the Context of Other Questions

A distractor may look good in isolation but receive a cue from a neighboring item. One stem reveals a definition needed to choose in another. Review the test as a whole, especially if AI generated a series from one template. Look for recurring key positions, identical structures in correct sentences, and systematic style differences. There is no need to mechanically equalize everything beyond recognition. Remove features that let students answer consistently without performing the subject action the course needs to assess.

For several test versions, check every set rather than only the original. Automatic replacement of numbers or names can break an explanation, create a second key, or make a wrong option obvious. Keep identifiers and revision history. After changing a candidate, mark which pilot conclusions concern the previous version. This discipline prevents transferring observed behavior of an old option to a new one with different difficulty, cues, or subject logic for participants.

Analyze the Pilot as Grounds for Review

Look at which alternatives participants choose and ask some to explain their decisions. An option nobody selected in a small group is not necessarily permanently unsuitable. The topic may be well learned or the sample too small. An option popular among strong students also needs substantive review: ambiguity, an uncertain key, or unplanned difficulty may be present. A statistical signal directs attention but does not replace checking the text, solution, and learning conditions under which this particular task was completed.

Keep actual response counts and group composition. Do not assign imaginary quality indicators to an unused task. AI can explain an analysis method, but cannot create actual student results without data. General requirements for checking generative outputs are described in the NIST profile. For editors, this means clearly separating a prediction of candidate behavior, pilot observation, and the final subject-specific decision. Each has a different origin and evidentiary strength for maintaining a test bank.

Make a Decision for Every Candidate

After review, retain a short status: accept, revise, or remove, with a reason. An accepted candidate needs a subject explanation, no second key under the stated conditions, and a clear role in the set. A revised candidate needs a new check. Do not keep dozens of rejected answers in the working test 'just in case.' They may accidentally return during another import or version mixing. Separate the verified item bank from the editorial candidate collection so status remains visible.

When the WeStudy LMS is used for learning, enter the approved set with verified explanations through a method available in your configuration. After transfer, check order, key, and formula display. If the system shuffles answers, check that feedback does not refer to an old letter or position. Do not assume automatic subject validation of distractors without confirmation of that capability. An item may be well prepared in a file and still need further acceptance after technical entry into the course.

The WeStudy LMS can support organizing prepared materials within its capabilities, while instructors retain substantive control. A useful AI result is a set of candidates that has passed transparent human selection. A good distractor helps reveal a boundary of understanding and provide subject-specific feedback. Its value comes from its connection to the task and behavior in a verified context. Keep reasons for decisions for the next cycle so a new author can develop the bank from established requirements rather than repeat selection from scratch.

Check Different Errors in a Second Numerical Example

A hypothetical measure increased from 20 to 25 units. If the question asks for relative growth against the initial value, the calculation gives 25 percent. Candidate 5 may represent the absolute difference, while 20 percent represents dividing the difference by the final value. Each alternative has its own explanation and is checked separately. However, these numbers do not form a good test without a clear statement of what to find. Authors must define the denominator and question before calling one answer correct and the others errors in student understanding.

If the initial values are themselves percentages, percentage change and percentage points must be distinguished. This is a different formulation requiring a corresponding explanation. Do not retain the old key after replacing 'units' with 'percent' merely because the numerical notation looks similar. Ask an editor to solve the task from scratch. AI sometimes transfers a solution template from the first example to the second without checking the changed meaning of the quantity. Checking must therefore cover the conditions, every number, and every explanation together, rather than an isolated word an author has just changed for variety.

Compare Two Candidates with the Same Explanation

AI sometimes creates different answers but assigns them the same cause. Check whether the faulty step actually yields both results. If not, at least one explanation is unreliable. If so, decide whether both alternatives are needed for the target knowledge. Duplication can narrow the set: several options may test one minor operation while another important boundary of understanding is overlooked. Assigning roles helps authors see why every answer is included and which subject-specific feedback participants receive after selecting it.

At the same time, do not demand that every candidate represent a psychologically unique error. In many topics, one click cannot reliably establish the cause of a choice. Keep wording cautious: the option is compatible with a particular faulty step, rather than proving the student thought that way. If deeper understanding of reasoning is needed, add a short explanation or a separate practical task. Distractors are part of assessment, not a universal tool for reading internal reasoning. This boundary should be clear to both instructors reviewing results and students receiving automatically prepared feedback.

Perform an Independent Check Without the Author's Explanations

Give a colleague only the full task and ask them to identify the answer, then explain what makes the other options unsuitable. Show your explanations only afterward. If the colleague sees a different role for a candidate, clarify the wording or interpret future student choices more cautiously. There is no need to settle every possible view by voting. Compare each interpretation with the conditions and subject source. If a genuine alternative remains acceptable, revise the task; if it violates an explicit condition, record that in the editorial decision so the next review does not begin with the same misunderstanding.

Check whether the author's explanation taught the colleague to see something absent from the text. Someone who already knows the key may easily fill in the required context. An independent pass helps reduce this effect. For a complex case, ask the reviewer to restate the question in their own words first. A discrepancy between the retelling and authorial intention may matter more than agreement on the selected answer. The colleague may have chosen the key coincidentally for another reason, so agreement alone does not establish that the set assesses the intended learning action and provides justified feedback afterward.

Check Technical Presentation Without Position Dependencies

If order can change, explanations should name answer content or the relevant action. 'Option B is wrong' can easily become false after shuffling. The same applies to 'the longest sentence' or 'the number below.' Check whether meaning is preserved on a small screen where answers wrap across lines. Do not infer equal conditions from the working file: the final interface may change the visibility of formulas, subscripts, or units. Acceptance requires reviewing the presentation participants will actually use during a learning attempt.

Keep the finished package in a controlled version. It includes the question, all answers, key, feedback, and subject notes for instructors. If one element changes, check dependencies in the rest. Separately mark which materials students see before and after answering. Incorrectly configured explanation display may reveal the key too early or leave students without the necessary analysis. This is not a problem in the distractor's text, but it changes how the entire item works. Subject editing and acceptance of final presentation must therefore be connected stages of one preparation process.

For every subsequent use, keep the acceptance date and the name of the person responsible for the package's subject review visible.

Sources

1. Bitew and colleagues: Learning to Reuse Distractors.

2. NBME: Item-Writing Guide.

3. University of Waterloo: Designing Multiple-Choice Questions.

4. NIST: Generative Artificial Intelligence Profile.