Evidence-centered design (ECD) is an orientation towards assessment development. It differs from conventional practice in several ways and consists of multiple activities. Each of these activities results in a set of useful documentation: domain analysis, domain modeling, construction of the assessment framework, and assessment…

Mixed-format tests containing both multiple-choice (MC) items and constructed-response (CR) items are now widely used in many testing programs. Mixed-format tests often are considered to be superior to tests containing only MC items although the use of multiple item formats leads to measurement challenges in the context of equating conducted under…

The Many-Facet Rasch (MFR) Model is traditionally used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR Model by examining the quality of ratings obtained from a…

This article demonstrates how textbooks differ in their description of the term "experimental unit". Advanced Placement Statistics teachers and students are often limited in their statistical knowledge by the information presented in their classroom textbook. Definitions and descriptions differ among textbooks as well as among different…

The many-faceted Rasch (MFR) model has been used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR model for examining the quality of ratings obtained from a standard…

Developing a cognitive model of task performance is an important and often overlooked phase in assessment design; failing to establish such a model can threaten the validity of the inferences made from the scores produced by an assessment (e.g., Leighton, 2004). Conducting think aloud interviews (TAIs), where students think aloud while completing…

This paper discusses issues related to the design and psychometric work for mixed-format tests --tests containing both multiple-choice (MC) and constructed-response (CR) items. The issues of validity, fairness, reliability and score consistency can be addressed but for mixed-format tests there are many decisions to be made and no examination or…

Mixed-format tests containing both multiple-choice and constructed-response items are widely used on educational tests. Such tests combine the broad content coverage and efficient scoring of multiple-choice items with the assessment of higher-order thinking skills thought to be provided by constructed-response items. However, the combination of…

This catalog lists research reports, research notes, and other publications available from the College Board's website. The catalog briefly describes research publications available free of charge. Introduced in 1981, the Research Report series includes studies and reviews in areas such as college admission, special populations, subgroup…

A special case of examinee choice, the Optional Essay Problem, is examined from the point of view of test equating. The Optional Essay Problem involves equating essay scores when the examinees are required to select an optional essay topic from a list of topics in addition to taking a mandatory test required of all examinees. The conditions that…

When test-takers are offered a choice of essay questions, some questions may be harder than others. If the test includes a common portion taken by all test-takers, an adjustment to the scores is possible. Previously proposed adjustment procedures disregard the test-makers' efforts to create questions of equal difficulty; these procedures tend to…

A method is proposed to assess the importance of differential item functioning (DIF) by estimating the largest possible fraction of the population in which DIF does not occur, or equivalently, the smallest possible portion of the population in which DIF may occur. The approach is based on latent class (C. C. Clogg, 1981) or mixture concepts, and…

The relationship between the multiple-choice and free-response sections of the Computer Science and Chemistry tests of the College Board's Advanced Placement program was studied. Confirmatory factor analysis showed that the free-response sections measure the same underlying proficiency as the multiple-choice sections for the most part. However,…

The 1999 Advanced Placement[R] (AP[R] Psychology Examination contains items drawn from 13 factors related to the study of psychology. This factor structure had not been explored previously. This study focuses on evaluating the fit of confirmatory factor analysis (CFA) models to examination items. Since examination items were dichotomous and…

Examined the connection between gender differences in examinees' familiarity, interest, and negative emotional reactions to items on the College Board's Advanced Placement Psychology Examination and the items' differential item functioning (DIF). Gender differences for a sample of 717 students for the 3 variables were substantially related to the…