The "2nd Annual WPI-UMASS-UPENN EDM Data Mining Challenge" required contestants to predict efficient testtaking based on log data. In this paper, we describe our theory-driven and psychometric modeling approach. For feature engineering, we employed the Log-Normal Response Time Model for estimating latent person speed, and the Generalized…

The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal…

Background/Context: Impact evaluations of schooling reforms in developing countries typically focus on tests of student achievement that are designed and implemented by researchers. Are these tests any good? What practical and principled guidance should researchers in the field follow? We aim to answer these questions. Test scores have, of course,…

The design and psychometric methodology of the National Assessment of Educational Progress (NAEP) is constantly evolving to meet the changing interests and demands stemming from a rapidly shifting educational landscape. NAEP has been built on strong research foundations that include conducting extensive evaluations and comparisons before new…

Rising inequality in the United States has raised concerns about potentially widening gaps in educational achievement by socioeconomic status (SES). Using assessments from LTT-NAEP, Main-NAEP, TIMSS, and PISA that are psychometrically linked over time, we trace trends in SES gaps in achievement for U.S. student cohorts born between 1961 and 2001.…

Rising inequality in the United States has raised concerns about potentially widening gaps in educational achievement by socio-economic status (SES). Using assessments from LTT-NAEP, Main-NAEP, TIMSS, and PISA that are psychometrically linked over time, we trace trends in achievement for U.S. student cohorts born between 1954 and 2001. Achievement…

In the absence of clear incentives, achievement tests may be subject to the effect of slipping where item response functions have upper asymptotes below one. Slipping reduces score precision for higher latent scores and distorts test developers' understandings of item and test information. A multidimensional four-parameter normal ogive model was…

In this essay, we describe the construction and use of the Cut-Score Operating Function in aiding standard setting decisions. The Cut-Score Operating Function shows the relation between the cut-score chosen and the consequent error rate. It allows error rates to be defined by multiple loss functions and will show the behavior of each loss…

Economists often use test scores to measure a student's performance or an adult's human capital. These scores reflect non-trivial decisions about how to measure and scale student achievement, with important implications for secondary analyses. For example, the scores computed in several major testing regimes, including the National Assessment of…

Early presidents of the American Educational Research Association were leaders in the testing movement. Their intentions were to improve education by means of testing, which included both IQ and achievement tests. Early measurement experts acknowledged in scholarly articles that IQ tests could not measure inherited ability of groups with vastly…

Background: Economists are making increasing use of measures of student achievement obtained through large-scale survey assessments such as NAEP, TIMSS, and PISA. The construction of these measures, employing plausible value (PV) methodology, is quite different from that of the more familiar test scores associated with assessments such as the SAT…

This article presents the author's observations on Neil Dorans's NCME Career Award Address: "The Contestant Perspective on Taking Tests: Emanations from the Statue within." He calls attention to some points that Dr. Dorans made in his address, and offers his thoughts in response.

Since its inception more than four decades ago, the National Assessment of Educational Progress (NAEP) has served as a key indicator of what the nation's students know and can do in academic subjects. NAEP assessments provide a mechanism for putting the achievements of students in all states on a common scale; the assessments also serve as…

"Learning progressions" are one of the most important assessment design ideas to be introduced in the past decade. In the United States, several committees of the National Research Council (NRC) have argued for the use of learning progressions as a means to foster both deeper mastery of subject-matter content and higher level reasoning…

Development of the Common Core State Standards (CCSS), and the creation of the Smarter Balanced Assessment Consortium (Smarter Balanced) and the Partnership for Assessment of Readiness for College and Careers (PARCC), changes the pattern of accountability testing. These changes raise the question: "How should NAEP's validity and utility be…