As the users of the scores from large-scale standardized tests have begun to demand more from tests than binary correct/incorrect answers can provide, test developers have turned increasingly to richer forms of assessment such as essays (Sireci & Zenisky, 2006), portfolios (Linn, Baker, & Dunbar, 1991), and even computer-displayed “cases” (Zenisky & Sireci, 2002). But these richer forms of assessment could not be scored sensibly as just correct/incorrect; it would be wasteful in the extreme to collapse all of the information in an examinee's essay to just a single bit of information. So, instead of a binary score, such complex tasks are typically scored polytomously. Such a scoring paradigm requires a more general psychometric scoring model to summarize components of tests, some of which might be scored in a binary way, others polytomously, and still others might be scored on a continuous scale.

Whether these forms of assessment, for the time that is spent constructing them, the amount it takes users to complete them, and the amount of time it takes to score them is more efficacious than just adding more multiple-choice items is an open empirical question and not one we shall address. However, this has been explored in part in Lukhele, Thissen, and Wainer (1994), and preliminary results suggest that from a purely information content perspective, multiple-choice items do pretty well. That is, if one wanted to just increase the reliability of the test such that the correlation between the observed score and the “true score” was maximized, multiple-choice items should play a significant part of tests.