This National Council on Measurement in Education (NCME) module is designed to help teachers understand test reliability requirements. Focus is on the reliability of scores from teacher-made achievement tests used for interpreting norm-referenced scores. Reliability, factors affecting scores, reliability estimation methods, reliability's relation…

The unfulfilled promise of criterion-referenced measurement is that it would permit valid inferences about what a student could and could not do. To come closest to achieving all that criterion-referenced testing originally promised, tests of higher item density, with more items per amount of domain, are required. (SLD)

Officials of the United States Department of Education conclude that the pervasive "above average" achievement reported by J. J. Cannell (1988) results from a combination of testing practices and procedures among school testing programs. Causative factors of this paradox, dubbed the "Lake Wobegon Phenomenon," are discussed.…

Representatives of the Riverside Publishing Company discuss the "Lake Wobegon Phenomenon," the reported "above average" achievement of most elementary schools nationwide that was discussed by J. J. Cannell (1988) of the Friends for Education. More appropriate use of normative comparisons and more complete reporting of test…

Recognizing the concerns raised by J. J. Cannell (1988), Science Research Associates (SRA) defends its norms. Trends in curriculum need to be monitored in addition to students' achievement levels. New norms are needed only when there is a significant shift in curriculum or student performance. (SLD)

Despite the skepticism of J. J. Cannell (1988) that commercial test scores reveal much about the real achievement level of students, there is ample evidence that at least some of the gains observed on standardized tests are legitimate. Standardized scores should be greeted with healthy skepticism when high, but taken very seriously when low. (SLD)

Issues in the measurement of subject matter proficiency, as defined by recognized subject matter scholars, are explored as they relate to the increasing use of teaching machines, programmed learning, and broadly conceived instructional systems. Criterion-referenced measurement is discussed. (SLD)

Beginning discussions and exploratory work on criterion-referenced measurement are reviewed in this commentary on the author's 1963 address to the American Educational Research Association on issues of measurement and the development of educational technology. Many problems foreseen at that time remain current. (SLD)

The term "criterion-referenced measurement" has been so frequently used that accumulated surplus meaning has sometimes swamped the fundamental concept. Stripping away extraneous interpretations affirms the contemporary relevance of the concept. Substituting the term "construct-referenced" underscores the need for better…

The instructional impact of criterion-referenced testing during the past three decades is reviewed, and a method of devising criterion-referenced tests is suggested that departs from the usual model. Adding verification along the lines of panel review that newly constructed test items do elicit appropriate information is suggested. (SLD)

The rise of criterion-referenced measurement from 1970 to 1984 was an important period in test development. Since 1984 there has been less research published on criterion-referenced measurement, but there is ample evidence that the concept remains important, especially in the authentic measurement and performance assessment movements. (SLD)

In response to J. J. Cannell's (1988) discussion of the reported "above average" achievement of the vast majority of public school systems nationwide, the nature of national norms is discussed. The public must be better informed about the interpretation of test results. (SLD)

Test specifications need to take a middle road so that they are not so general that instructional targeting is impossible and not so specific that only unitary assessment targets are fostered. Using illustrative sample items provides a good means of communicating the items eligible to be tested. (SLD)

Cohen and Hyman's response contains several misunderstandings of the original article by Mehrens and Kaminski. One frequently wishes to make inferences to a domain from a test, but teaching a specific performance and testing for that performance does not allow for a domain inference. (SLD)

Some unfinished issues relating to achievement test theory that seemed implicit in the basic idea of criterion-referenced testing are reviewed, recognizing their importance in current studies of authentic assessment and performance-based tests. The future of performance-based evaluation is explored. (SLD)