Background/Context: Pass/fail standardized exams frequently selectively rescore failing exams and retest failing examinees. This practice distorts the test score distribution and can confuse those who do analysis on these distributions. In 2011, the Wall Street Journal showed large discontinuities in the New York City Regent test score…

Results obtained by the Kudar-Richardson formula (20) adapted for use with R-KW scoring are compared with three other reliability formulas. Based on parallel tests administered at the same sitting the KR (20) estimates are compared with alternate-form correlations and with odd-even correlations adjusted by the Spearman-Brown prophecy formula.…

Seventeen statistics for measuring the reliability of criterion-referenced tests were critically reviewed. The review was organized into two sections: (1) a discussion of preliminary considerations to provide a foundation for choosing the appropriate category of "reliability" (threshold loss function, squared-error loss-function, or…

Fabricated test protocols were used to study how effectively examiners agree in scoring ambiguous WISC-R responses. The results suggest that, even with the improved WISC-R manual, scoring remains a difficult and challenging task. (Author)

To date a theoretical basis has not been developed for determining changes in reliability when score points from random guessing are eliminated and those from non-randon guessing are retained. This paper presents a derivation of an expression for the reliability coefficient which displays the effect of deleting score components due to random…

A new formula for the correction for chance success due to guessing was advanced and investigated, the mathematical solution of which has the property of equating the scores of fast and slow examinees of equal ability. (Author/JKS)

Six different approaches to scoring test data, including number right, correction for guessing, and answer-until-correct, were investigated using Monte Carlo techniques. Modes permitting multiple response showed higher internal consistency, but there was little difference among modes for a validity measure. (JKS)

The 1966 SVIB scoring keys were investigated and found to contain an average of 214 fewer items per occupational scale than the 1938 scoring keys. The shorter scales are less reliable than the longer scales on the 1966 SVIB. (Author)

Supports arguments against general use of change scores and recommends the Lord/McNemar estimates of true change. Provides a numerical example illustrating the reliability problem and the problem of the prediction of true change from various linear composites of initial and final measures. (Author)

Investigates the reliability and validity of scores yielded from a new scoring formula. (Author/DEP)

In this paper, an attempt has been made to synthesize some of the current thinking in the area of criterion-referenced testing as well as to provide the beginning of an integration of theory and method for such testing. Since criterion-referenced testing is viewed from a decision-theoretic point of view, approaches to reliability and validity…

A potentially valuable measure of overconfidence on probabilistic multiple-choice tests was evaluated. The measure of overconfidence was based on probabilistic responses to nonsense items embedded in a vocabulary test. The test was administered under both confidence response and conventional choice response directions to 208 undergraduate…

A study involving 105 sixth-graders examined three aspects of technical adequacy with respect to two general outcome measures in mathematics: the effects of aggregating scores and correcting for random guessing on reliability and validity and the extent to which the measures were sensitive to changes in performance. (Contains references.)…

An investigation was conducted to determine the effects of response-category weighting and item weighting on reliability and predictive validity. Response-category weighting refers to scoring in which, for each category (including omit and "not read"), a weight is assigned that is proportional to the mean criterion score of examinees selecting…

The Ohio Vocational Achievement Tests are specially designed instruments for use by teachers, supervisors, and administrators to evaluate and diagnose vocational achievement for improving instruction in secondary vocational programs at the 11th and 12th grade levels. This guide explains the Ohio Vocational Achievement Tests and how they are used.…