Scoring procedures for the constructed-response (CR) items in large-scale mixed-format educational assessments often involve checks for rater agreement or rater reliability. Although these analyses are important, researchers have documented rater effects that persist despite rater training and that are not always detected in rater agreement and…

Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational Progress (NAEP) data mining competition. The…

This study examines the practical implications of using state tests to measure student achievement in impact evaluations that span multiple states and grades. In particular, the study examines the sensitivity of impact findings to (1) the type of assessment used to measured achievement (state tests or an external assessment administered by the…

In the summer of 2011, the Tennessee Department of Education contracted with the National Institute for Excellence in Teaching (NIET) to provide a four-day training for all evaluators across the state. NIET trained more than 5,000 evaluators intensively in the state model (districts using alternative instruments delivered their own training).…

Developing scoring rubrics to evaluate student work was studied, concentrating on the use of intermediate points in rating scales. How scales that allow for intermediate points between defined categories should be constructed and used was explored. In the recent National Assessment of Educational Progress (NAEP) visual arts field test, researchers…

Rigorous research provides information that will allow other schools and districts to select interventions that have a scientifically based track record of effectiveness. All Striving Reader grants include the mandate to evaluate literacy intervention(s) targeted to adolescents who are reading significantly below grade level. Although all studies…

This study involved the selection and adaptation of a writing assessment procedure for teachers and researchers in the Duval County Public Schools (Florida) to use in assessing changes in writing ability among elementary grade students. Through a review of the literature, four writing assessment procedures (analytic, holistic, focused holistic,…

The National Assessment Governing Board (NAGB) of the National Assessment of Educational Progress (NAEP) adopted mathematics achievement standards on May 11, 1991 that will be used to report results on the 1990 assessment of mathematics. NAEP achievement levels are descriptions, based on informed judgments, of what students should know and be able…

Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are…

In September 1991, the National Assessment Governing Board (NAGB) announced standards for basic, proficient, and advanced achievement in mathematics and reported that few American students had reached these standards. Expert reviewers noted technical problems with the NAGB approach and questioned its results. In this report, the NAGB…