We provide evidence of validity for a newly developed diagnostic competence model of operation sense, by both (a) describing the theoretically substantiated development of the competence model in close association with its use within a large-scale formative assessment and (b) providing empirical evidence for the theoretically described cognitive…

This paper provides a history of ETS's role in developing assessment instruments and psychometric procedures for measuring change in large-scale national assessments funded by the Longitudinal Studies branch of the National Center for Education Statistics. It documents the innovations developed during more than 30 years of working with…

Two approaches to criterion-referenced measurement are described and contrasted--domain-referenced testing and mastery testing. This paper is organized according to ten issues or stages in test construction: (1) content domain specification; (2) item construction; (3) item domain specification; (4) item analysis; (5) item selection; (6) parallel…

Procedures for establishing standards and determining the number of items needed in criterion-referenced measures are reviewed. The discussion of setting a passing score is organized around five factors: performance of others, item content, educational consequences, psychological and financial costs, and measurement error. Classical test theory,…

This document describes the development and application of a model for generating a core reading vocabulary screening test based on rank frequency distributions derived from a quarter of a million word corpus of 500 word samples embracing commerce, English, home economics, industrial education, mathematics, science, and social studies. The word…

Two techniques for writing achievement test items to accompany instructional materials are contrasted: writing items from statements of instructional objectives, and writing items from semi-automated rules for transforming instructional statements. Both systems resulted in about the same number of faulty items. (Author/JKS)

This paper discusses the validity of test preparation practices in the context of large-scale norm-referenced and criterion-referenced tests. It presents a framework for conceptualizing the various aspects of test preparation that includes the basic premises underlying the validity of the practices. Five areas of test preparation are outlined: (1)…

Binomial models for domain-referenced testing are compared, emphasizing the assumptions underlying the beta-binomial model. Advantages and disadvantages are discussed. A proposed item sampling model is presented which takes the effect of guessing into account. (Author/CM)

A unified framework for validating criterion-referenced test (CRT) interpretation is proposed. Using functional literacy as an illustration, the instructional outcomes assessed are called achievement constructs, and described in psychological and behavioral terms. The proposed construct validation methods can yield tests more closely linked to…

The purpose of this study was to examine the assembly and system models of reading with respect to the development of phoneme-grapheme association skills in normal and disabled readers. (Author)

A battery of criterion referenced job task performance tests (JIPT) for typical electronic maintenance activities were developed. The construction of a battery of such tests together with an appropriate scoring for reporting the results is detailed. The development of a Test Administrators Handbook also is described. This battery is considered to…

On the basis of the characteristics of criterion-referenced tests, the contribution of German research to the development and application of criterion-referenced tests is discussed. (PN)

An operational framework and a set of empirical procedures for building achievement tests sensitive to the instructional effects of school programs are presented. Procedures are illustrated using data from a comparative study of alternative item formats for a criterion-referenced test. Important differences in instructional sensitivity are shown.…

This report summarizes the research findings of a four year contract investigating the applicability of item response theory and tailored testing to criterion-referenced measurement. Six major areas were studied on the project: (1) techniques for forming unidimensional item sets; (2) techniques for calibrating items; (3) item parameter linking…

The development of a test design methodology used to construct a criterion-referenced System Achievement Test (CR-SAT) for selected Naval enlisted classification (NEC) in the Strategic Weapon System (SWS) of the United States Navy is described. Subject matter experts, training data analysts and educational specialists developed a comprehensive…