Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing…

Rater training is an important part of developing and conducting large-scale constructed-response assessments. As part of this process, candidate raters have to pass a certification test to confirm that they are able to score consistently and accurately before they begin scoring operationally. Moreover, many assessment programs require raters to…

The evidence-centered design framework was used to create a special education teacher observation system, Recognizing Effective Special Education Teachers. Extensive reviews of research informed the domain analysis and modeling stages, and led to the conceptual framework in which effective special education teaching is operationalized as the…

Neil Dorans has made a career of advocating for the examinee. He continues to do so in his NCME career award address, providing a thought-provoking commentary on some current trends in educational measurement that could potentially affect the integrity of test scores. Concerns expressed in the address call attention to a conundrum that faces…

The Joint Committee on Testing Practices has completed the "Code of Fair Testing Practices in Education," which is meant for the public and focuses on the proper use of tests in education--admissions, educational assessment and diagnosis, and student placement. The Code separately addresses test developers' and users' roles. (SLD)

A summary of the writing assessment programs reviewed in this journal is presented. The problems inherent in the programs are outlined. A coordinated research program on major problems in writing assessment is proposed as being beneficial and cost-effective. (DWH)

Test development procedures and six methods for determining cut-off scores are briefly described. An alternate method, appropriate when the test developer also determines the cut-off score, is suggested. Unlike other methods, the standard is set during the test development stage. Its computations are intelligible to nonstatistically-oriented…

A newly developed performance standard-setting procedure, termed iterative judgmental policy capturing (JPC), is applicable to assessments composed of distinct multidimensional exercises. The procedure is described, and results are reported from the application of JPC in a study involving a panel of 20 teachers and 6 performance exercises. (SLD)

Conceptual, practical, and logistical issues in the development of science performance assessments (SPAs) are discussed. The conceptual framework identifies task, response format, and scoring system as components, and conceives of SPAs as tasks that attempt to recreate conditions in which scientists work. Developing SPAs is a sophisticated effort…

Generalizability of performance-based assessment scores across raters and tasks is examined, focusing on implications of generalizability analyses for specific uses and interpretations of assessment results. Although it seems probable that assessment conditions, task characteristics, and interactions with instructional experiences affect the…

Context-dependent item sets, containing a subset of test items related to a passage or stimulus, are discussed. A brief review of methods for developing item sets reveals their potential for measuring high-level thinking. Theories and technologies for scoring item sets remain largely experimental. Research needs are discussed. (SLD)

In every instance in the process of constructing and using a test, the microcomputer can aid the classroom teacher. However, the teacher will not apply the microcomputer to classroom testing without added training both in classroom testing and in using the microcomputer. (BW)

Incidents affecting the implementation of large-scale testing programs are described to illustrate associated problems. Issues addressed include creation of test materials, preparation of answer documents, transportation of test materials, scoring and analysis of tests, and dissemination and utilization of test results. (TJH)

The new Florida Master Teacher Program in which the state provides bonuses directly to qualified teachers is described. Three measurement issues in implementing the program are discussed: (1) evaluating a teacher's classroom performance; (2) evaluating a teacher's subject area knowledge; and (3) combining scores to determine which teachers…

Literature related to the multiple true-false (MTF) item format is reviewed. Each answer cluster of a MTF item may have several true items and the correctness of each is judged independently. MTF tests appear efficient and reliable, although they are a bit harder than multiple choice items for examinees. (SLD)