This article discusses critical methodological design decisions for collecting, interpreting, and synthesizing empirical evidence during the design, deployment, and operational quality-control phases for automated scoring systems. The discussion is inspired by work on operational large-scale systems for automated essay scoring but many of the…

Assessment engineering is a new way to design and implement scalable, sustainable and ideally lower-cost solutions to the complexities of designing and developing tests. It represents a merger of sorts between cognitive task modeling and engineering design principles--a merger that requires some new thinking about the nature of score scales, item…

The purpose of this report is to describe the technical qualities of the 2018-2019 operational administration of the English language arts/literacy (ELA/L) and mathematics summative assessments in grades 3 through 8 and high school. The ELA/L assessments focus on reading and comprehending a range of sufficiently complex texts independently and…

The purpose of this report is to describe the technical qualities of the 2018-2019 operational administration of the English language arts/literacy (ELA/L) and mathematics assessments in grades 3 through 8 and high school. New Meridian, in coordination with multiple states and vendors, developed an alternate form of the summative assessment to…

At the heart of most assessments lies a set of questions, and those who write them must achieve "two" things. Not only must they ensure that each question elicits the kind of performance that shows how "good" pupils are at the subject, but they must also ensure that each mark scheme gives more marks to those who are…

Automated scoring of constructed responses is already operational in several testing programmes. However, as the methodology matures and the demand for the utilisation of constructed responses increases, the volume of automated scoring is likely to increase at a fast pace. Quality assurance and control of the scoring process will likely be more…

There are few available instruments that assess students' perceptions of university teaching and learning environments and can be used to provide diagnostic feedback, at the degree or program level, on the quality of the environment. This article describes the development, testing and application, for quality assurance purposes, of a suitable…

Drawing on the experience of Oregon's new-teacher evaluation system, describes teacher work-sample methodology and evidence for its reliability and validity in evaluating the performance of prospective teachers. This is a methodology that responds to the challenge issued by the National Commission on Teaching and the America's Future challenge.…

Quality measures were originally developed for research aimed at describing the settings that children spend time in and identifying the characteristics of these environments that contribute to children's development. They were also developed to guide improvements in practice. Increasingly, however, measures of quality are being used for further…

A taxonomy of potential values conflicts in evaluations is presented. Basis is the distinction of types of conflict (aims vs. means conflicts; qualitative vs. quantitative conflicts). Six areas of values are discussed: the ethical, methodical, social and interactive, legal, economic, and personal values. The taxonomy of conflicts between these…