Use of flawed multiple-choice items by the New England Journal of Medicine for continuing medical education

nejm2.gif It will be interesting to see what the editors of the New England Journal of Medicine think of this study:

Alex S. Stagnaro-Green  and Steven M. Downing. Use of flawed multiple-choice items by the New England Journal of Medicine for continuing medical education.  Med Teach 2006; 28 (6):566-568. 

Abstract: Physicians in the United States are required to complete a minimum number of continuing medical education (CME) credits annually. The goal of CME is to ensure that physicians maintain their knowledge and skills throughout their medical career. The New England Journal of Medicine (NEJM) provides its readers with the opportunity to obtain weekly CME credits. Deviation from established item-writing principles may result in a decrease in validity evidence for tests. This study evaluated the quality of 40 NEJM MCQs using the standard evidence-based principles of effective item writing. Each multiple-choice item reviewed had at least three item flaws, with a mean of 5.1 and a range of 3 to 7. The results of this study demonstrate that the NEJM uses flawed MCQs in its weekly CME program.    DOI Link 

Table 1. Item Flaws in 40 NEJM CME test items.

Type of flaw

Number of flaws

Percentage of 40 items flawed

1. Verbatim text



2. Unfocused stem



3. Window dressing



4. Unequal option length



5. Negative options



6. Clues to correct answer



7. Negative stem



8. Heterogeneous options






1.  Verbatim text flaw= much of the MCQ text is identical to the article. To answer correctly, the participant is not required to understand the information, but need only locate the identical text in the article.
2.  Unfocused stem flaw= no question is posed by the lead-in or stem of the item. The participant must read all of the options in order to understand what question is being asked.
3.  Window dressing flaw= the MCQ has excessive verbiage that is irrelevant to the question asked or the construct being assessed.
4.  Unequal option length flaw= the correct option is appreciably longer than the alternative incorrect options. This provides a testwise clue to the correct answer, since true statements are usually longer than false statements.
5.  Negative options flaw= the options contain negative words. For example, “The article did not conclude…”.
6.  Clues-to-correct answer flaw= the MCQ provides a direct testwise clue to the correct answer. For example, the use of ‘always’, ‘never’ or ‘absolutely’ in an option identifies that option as incorrect.
7.  Negative-stem flaw= the stem includes negatives such as not or except.
8.  Heterogeneous-options flaw= the options are not homogenous in content and/or grammatical structure.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

%d bloggers like this: