The "2nd Annual WPI-UMASS-UPENN EDM Data Mining Challenge" required contestants to predict efficient testtaking based on log data. In this paper, we describe our theory-driven and psychometric modeling approach. For feature engineering, we employed the Log-Normal Response Time Model for estimating latent person speed, and the Generalized…

The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal…

Background: Large-scale survey assessments have been used for decades to monitor what students know and can do. Such assessments aim at providing group-level scores for various populations, with little or no consequence to individual students for their test performance. Students' test-taking behaviors in survey assessments, particularly the level…

Techniques to identify the degree to which a response pattern is unusual and the pattern's relationship to examinee background are presented. Sato's caution index and modified caution index use clusters of items for comparisons of observed performance outcomes and do not require the use of item response theory. Correlation between the indices was…

Racial attitude questions from national surveys such as the National Assessment of Educational Progress (NAEP) are being used to show that progress has been made toward interracial harmony in this country. However, there is reason to worry about conclusions from such measures, because they may indicate only progress on learning the socially…

Assessment exercises (items) in three different formats--multiple-choice with an "I don't know" (IDK) option, multiple-choice without the IDK, and open-ended--were placed at the beginning, middle and end of 45-minute assessment packages (instruments). A balanced incomplete blocks analysis of variance was computed to determine the biasing…

Over the past decade, developers of the National Assessment of Educational Progress (NAEP) have changed substantially the mix of item types on the NAEP assessments by decreasing the numbers of multiple-choice questions and increasing the numbers of questions requiring short- or extended-constructed responses. These changes have been motivated…

The multiple-choice science exercises used by the National Assessment of Educational Progress include an "I Don't Know" (IDK) alternative to estimate more accurately knowledge of groups of respondents. Group percentages of IDK responses were examined and compared with correct responses to see if the IDK introduces bias. Variance common…

The National Assessment of Educational Progress (NAEP) focuses on providing information on what demographic subgroups of America's students know and can do. Because the NAEP does not report for individual students, it could be classified by some students as a low-risk test. Consequently, some students may lack proper motivation for giving their…

The 168 items selected for administration to 7,905 17-year olds in 1976-77, were developed from existing National Assessment items and from existing sets of life/coping skills. Skills designated as basic included: career development; citizenship; community resource utilization; consumer protection; family management; health maintenance; and…