This paper follows a line of logical argumentation to claim that what Samuel Messick conceptualized about construct validation has probably been misunderstood by some educational policy makers, practicing educators, and classroom teachers. It argues that, while Messick's unified theory of test validation aimed at (a) warning educational…

A behavioral domain is well-defined when it is clear to both test developers and test users which categories of performance should or should not be considered for potential test items. Only those tests that are keyed to well-defined domains meet the definition of criterion-referenced tests. The greatest proliferation of criterion-referenced tests…

Questions about the appropriateness of the Nelson-Denny Reading Test have led to consideration of the use of a criterion referenced test for assessing students' reading abilities before and after each semester of developmental reading instruction. In constructing such a test, a number of steps should be followed. The first step is to clarify the…

Confusion in the terminology used in criterion-referenced measurement specifications and development and standard setting and the attendant role of cut-off scores are shown to need practical clarification through psychometric research on test applications and consequences. (CM)

These studies in test adequacy focus on two problems: procedures for estimating reliability, and techniques for identifying ineffective distractors. Fourteen papers are presented on recent advances in measuring achievement (a response to Molenaar); "an extension of the Dirichlet-multinomial model that allows true score and guessing to be…

Some unfinished issues relating to achievement test theory that seemed implicit in the basic idea of criterion-referenced testing are reviewed, recognizing their importance in current studies of authentic assessment and performance-based tests. The future of performance-based evaluation is explored. (SLD)

The potential of a cognitive model task analysis scheme (CMS) that specifies features of test problems shown by research to affect performance is explored. CMS describes the general skill area and the generic task or problem type. It elaborates features of the problem situation and required responses found by research to influence performance.…

Presents major stages in design and development of criterion referenced tests (CRT) with emphasis on differences between CRT construction and norm-referenced test construction. Discussion covers test interpretation; test theory; preparation for test construction (hierarchical analysis, item type selection, and choosing number of items); test…

Four discrimination indices were compared, using score distributions which were normal, bimodal, and negatively skewed. The score distributions were systematically varied to represent the common circumstances of a military training situation using criterion-referenced mastery tests. Three 20-item tests were administered to 110 simulated subjects.…

Standardized achievement tests are often misused as indicators of a school's quality or effectiveness relative to other schools. This is an incorrect use because it ignores variation among schools in student abilities, family support of education, student mobility, and other factors. People also misuse tests because they impute to them more…

The feasibility of criterion referenced testing is held to be dependent on the tenability of two postulates: (1) that bias can be controlled in a principled manner from one test to the next; and (2) that one mental process measured by such tests may lawfully interact with another. Without the first postulate, criterion scores could not be…

During a three-year project (1977-1980) on computerized adaptive achievement testing, item characteristic curve theory (ICC) and adaptive testing strategies designed almost exclusively for ability testing were applied to achievement testing. Adaptive techniques substantially reduced test length without reducing quality, when applied to three…

This report describes the development, operation, maintenance, and future prospects of the item banks pioneered by the Portland (Oregon) School District. At the time of this report, there were 3,500 mathematics, 2,200 reading, and 2,300 language usage items calibrated under the fixed parameter model of item response theory (IRT) for Grades 3-8.…

A procedure is demonstrated for constructing tables showing, for each score on a commercial reading achievement test, the percentage of real-world materials that the testee is likely to comprehend with at least a criterion level of proficiency, the percentages of students in a local or national sample who can competently comprehend a given…

This report was designed to respond to two major methodological shortcomings in the item bias literature: (1) misfitting test models; and (2) the use of significance tests. Specifically, the goals of the research were to describe a newly developed method known as the "plot method" for identifying potentially biased test items and to…