This paper extends the theory of conditional covariances to polytomous items. It has been mathematically proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, is positive if the two items are dimensionally homogeneous and negative…

Recent years have seen widespread adoption of automatic grading of physics homework problems. Although first-generation homework management systems provide feedback and assign scores based only on the answers, problem-solving performance on exams seems to be no worse for students using these systems than for those who receive detailed comments…

This report describes a statistically designed experiment that was carried out in an operational setting to determine the contributions of different sources of variation to the unreliability of scoring. The experiment made novel use of partially balanced incomplete block designs that facilitated the unbiased estimation of certain main effects…

The dynamic linear model or Kalman filtering model provides a useful methodology for predicting the past, present, and future states of a dynamic system, such as an object in motion or an economic or social indicator that is changing systematically with time. Recursive likelihood methods for adaptive Kalman filtering and smoothing are developed.…

The project explored the assessment of vocational decision-making capacities of handicapped students within the educational system, specifically the Decision-Making Interview (DMI) instrument. The adaptability, reliability, validity, and utility of the DMI with handicapped secondary students were investigated. Under an instrument adaptation phase…

Intended for administrators and policymakers as well as teachers, this digest explores approaches to, problems with, and trends in large scale writing assessment. The digest first compares direct and indirect assessment approaches, then examines three approaches to scoring: holistic, primary trait, and analytic. It then discusses issues and…

This study describes preliminary work on the development of a scale to assess attitude toward public drinking involvement. A sample of 46 students at a west-coast university participated in the study. Respondents completed a questionnaire containing a measure of Attitude Toward Public Drinking Involvement Scale (ATPDIS) and various validity…

An alternative method of identifying effective schools, based on the concept of effective schools as statistically atypical, was tested. This paper investigates the issue of consistency of the identification of what is termed "exceptional" schools. The following tests were administered to all first through fourth graders in 431 schools…

Three studies assessed the psychometric characteristics of the Direct Standard Setting Method (DSSM). The Angoff technique was also used in each study. The DSSM requires judges to consider an examination 10 items at a time and determine the minimum items in that set a candidate should answer correctly to receive the credential. Nine judges set a…

Cut scores obtained for licensure tests using different judgmental methods of standard setting (holistic, test blueprint, Angoff, and modified Angoff) were compared. Nineteen educators and practitioners participated in this study as judges. Pre- and post-test feedback (feedback of total- and low-group item p-value) ratings were obtained under the…

The objective description and identification of variables that meaningfully distinguish reasoning skills couched in written discourse was studied, comparing scores obtained from different perspectives on the same writing samples. A total of 406 writing samples on 2 topics by 203 students who had taken the Graduate Record Examinations, mostly…

A new approach to polychotomous scoring of test items, similar to "max-alpha" scaling (MAS) and known as polyweighting, has been developed. Unlike MAS, this new method of polychotomous scoring provides scoring weights for a given item that are independent of the difficulty of other items in the analysis. Moreover, the scoring weights are…

Confirmatory multidimensional item response theory (CMIRT) was used to assess the structure of the Graduate Record Examination General Test, about which much information about factorial structure exists, using a sample of 1,001 psychology majors taking the test in 1984 or 1985. Results supported previous findings that, for this population, there…

Procedures for correcting for guessing and for assessing partial knowledge (correction-for-guessing, three-decision scoring, elimination/inclusion scoring, and confidence or probabilistic scoring) are discussed. Mean scores and internal consistency reliability estimates were compared across three administration and scoring procedures for…