This paper reports on a standard-setting method designed for complex performance assessments with multiple performance categories. The method studied, the Analytical Judgment Method, involves panelists' making analytical classification decisions for each of the test's components individually. It also allows for discussion and reconsideration of…

Three descriptive statistics (difficulty, discrimination, and reliability) of multiple-choice (MC) test items were compared to those of a new (1980s) format of machine-scored questions. The new method, answer-bank multi-digit testing (MDT), uses alphabetized lists of up to 1,000 alternatives and approximates the completion style of assessment…

Constructed-response formats are desired for measuring complex and dynamic response processes that require the examinee to understand the structures of problems and micro-level cognitive tasks. These micro-level tasks and their organized structures are usually unobservable. This study shows that elementary graph theory is useful for organizing…

Instructions for the administration of the Opinions about Deaf People Scale are given. This scale measures the beliefs of hearing adults about the capabilities of deaf adults. The somewhat ambiguous title of the instrument is designed to avoid leading respondents to respond in socially desirable ways. The instrument is based on misconceptions…

Theoretical issues underlying the development of scales of language proficiency are examined. First, a brief classification of scale types is presented, and problems identified in them and advantages they offer are outlined. Discussion then moves to the issues of describing and measuring language proficiency. In this section, behaviorally-based…

The present study examines the influence of judges' item-related knowledge on setting standards for competency tests. Seventeen judges from different professions took a 122-item teacher-certification test in economics while setting competency standards for the test using the Angoff procedure. Judges tended to set higher standards for items they…

The International Association of Personnel Management Assessment Council (IPMAAC) is a section of the International Association of Personnel Management devoted to individuals involved in professional level public personnel assessment. Author-generated summaries/outlines of papers presented at the IPMAAC's 1986 conference are provided. The…

As part of a study of teacher evaluation of student replies to open-ended questions, a second question--the best method of determining interrater reliability--was examined. The standard method, the Pearson Product-Moment correlation, overestimated the degree of match between researchers' and teachers' scoring of tests. The simpler percent…

Orange County (California) students in grades 9 and 10 compete in an annually held series of 10 competitive events measuring academic strengths. These events include tests in grammar and literature, fine arts, mathematics, science, social science, study skills, and a super quiz--a team event held before a large audience. In addition, there are…

An adaptation of an impressionistic scoring method developed for use on student placement in an English as a second language (ESL) program is described and a correlational study of its reliability is presented. The method was chosen because of its efficiency and apparent reliability. The original emphasis of organization, length, and content of…

Observational ratings of student clinical performance are influenced by factors other than the quality of the performance. Individual raters may be more stringent or lenient than their colleagues. In this medical school setting, multiple raters evaluated each student. To reduce the influence of "error" due to differences among raters, each rater…

This bulletin has three purposes: (1) to discuss the variety of factors which should be considered in the development of a local testing program, providing a basic understanding of what is necessary to implement and maintain a testing program; (2) to suggest several of the significant elements related to the development and maintenance of a…

This paper describes the scoring of the upper levels of the Career Ladder system for teachers in the Tennessee Career Ladder Evaluation Systems. The instruments used to evaluate teachers include classroom observation, dialogue with the teacher, a peer questionnaire, student questionnaires, a professional skills test, professional development and…

The purpose of this book is to make it possible for measurement specialists to solve practical testing problems through the use of item response theory (IRT). The topics, organization, and presentation are those used in a 4-week seminar held each summer for the past several years. The material is organized to facilitate understanding; all related…

This text was written to help the reader acquire a base of knowledge about classical psychometrics and to integrate new ideas into that framework of knowledge. The material is organized into five units: (1) introduction to measurement theory; (2) reliability; (3) validity; (4) item analysis in test development; and (5) test scoring and…