In generalizability theory studies in large-scale testing contexts, sometimes a facet is very sparsely crossed with the object of measurement. For example, when assessments are scored by human raters, it may not be practical to have every rater score all students. Sometimes the scoring is systematically designed such that the raters are…

The purpose of this report is to describe the procedure for revising the ReadiStep™ score scale using the field trial data, and to provide technical information about the development of the new ReadiStep scale score. In doing so, this report briefly introduces the three assessments--ReadiStep, PSAT/NMSQT®, and SAT®--in the College Board Pathway…

There are relatively few examples of quantitative approaches to quality control in educational assessment and accountability contexts. Among the several techniques that are used in other fields, Shewart charts have been found in a few instances to be applicable in educational settings. This paper describes Shewart charts and gives examples of how…

The focus of this presidential address is on the contemporary treatment of reliability and validity in educational assessment. Highlights on reliability are provided under the classical true-score model using tools from latent trait modeling to clarify important assumptions and procedures for reliability estimation. In addition to reliability,…

A dramatic shift in research priorities has recently produced a large number of ambitious randomized trials in K-12 education. In most cases, the aim is to improve student academic learning by improving classroom instruction. Embedded in these studies are theories about how the quality of classroom must improve if these interventions are to…

The Bologna Declaration brought reforms into higher education that imply changes in teaching methods, didactic materials and textbooks, infrastructures and laboratories, etc. Statistics and mathematics are disciplines that traditionally have the worst success rates, particularly in non-mathematics core curricula courses. This research project,…

Preffi 2.0 is an evidence-based Dutch quality assessment instrument for health promotion interventions. It is mainly intended for both planning and assessing one's own projects but can also be used to assess other people's projects (external use). This article reports a study on the reliability of Preffi as an external quality assessment…

Using data from the final two rounds of the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B), a longitudinal study begun in 2001, this First Look provides a snapshot of the demographic characteristics, reading and mathematics knowledge, fine motor skills, school characteristics, and before- and after-school care arrangements of the cohort…

The stability of scores on 2 types of performance assessments, an observed hands-on investigation and a notebook surrogate, was investigated for 29 sixth graders on 2 occasions. Results indicate that student performance and procedures changed and that generalizability across occasions was moderate. Implications for assessment are discussed. (SLD)

This study provides empirical evidence about the sampling variability and generalizability (reliability) of a statewide performance assessment for grade six. Results for 600 students at individual and school levels indicate that task-sampling variability was the major source of measurement error. Rater-sampling variability was negligible. (SLD)