Describes method of indexing interobserver reliability from data relating to mutually exclusive nominal categories. Program computes reliability estimates from nominal data collected in a time interval manner. A specified critical time interval, order of responses, and number of responses are important factors in computing amount of agreement…

A 9-item questionnaire was given to registering students for fall and spring 1969. For easier handling, its form was changed for the next semester. By accident, the two versions were handed out and 687 students answered both in quick succession. The staff decided to compare the two forms to see if the answers differed. Two questions were…

An equation, introduced by Goodman and Kruskal for obtaining a reliability measure of one item was expanded. This formula determines inter-rater reliability for a series of items across many subjects. The statistic that results is easily interpreted and in many ways is analogous to the conventional reliability for quantitative data. (Author/BJG)

A German version of the "International Classification of Impairments, Disabilities, and Handicaps" was tested in respect to interrater reliability both on the method of relative agreement between raters and on the measure "kappa." Unsatisfactory reliability was found. (Author/DB)

This paper delineates the factors that influence the joint impact of attenuation and range restriction on the magnitude of the correlation coefficient. A theoretical framework for the generation of appropriate correction formulas is established. (TJH)

This paper suggests that although data from a homogenous sample might yield less reliable scores than did an inducted sample, these data should not be discarded until further examination of the data is conducted. The paper presents two statistics for monitoring data homogeneity and one statistic for correcting alpha when homogeneity is large. The…

Greater understanding and use of confidence intervals is central to changes in statistical practice (G. Cumming and S. Finch, 2001). Reliability coefficients and confidence intervals for reliability coefficients can be computed using a variety of methods. Estimating confidence intervals includes both central and noncentral distribution approaches.…

The scores on two distinct tests (e.g., essay and objective) are often combined into a composite score, which is used to make decisions. The validity of the observed composite can sometimes be evaluated relative to a separate criterion. In cases where no criterion is available, the observed composite has generally been evaluated in terms of its…

Presents ...a method for assessing the level of difficulty of reading material in the 7 to 15 year range, together with a supplementary method for dealing with material below the 7 year level." (Author)

Coefficient Kappa is generally defined in terms of procedures of computation rather than in terms of a population. Here a population definition is proposed. Factors influencing the magnitude of Kappa are identified. Strategies to improve reliability are proposed, including that of combining multiple unreliable diagnoses. (Author/CTM)

A Monte Carlo evaluation of 30 internal criteria for cluster analysis was conducted using four hierarchical clustering techniques. The results indicated that a subset of internal criteria was identified which appear to be valid indices of correct cluster recovery. (Author/JKS)

This study explores a strategy for improving the stability of variance component estimates when only small samples are available, using a series of small, less complex generalizability (G) study designs as a surrogate for a single large design. (Author/BW)

In at least two situations, use of the Spearman-Brown prophesy formula yields overestimates of the interobserver reliability of composite scores. The appropriate formulas for estimating the interobserver reliability of composite scores, as well as efficient means of estimating the elements in these formulas are presented. (Author)

Reliability generalization (RG) is a measurement meta-analytic method used to explore the variability in score reliability estimates and to characterize possible sources of the variance. Summarizes some RG considerations, and suggests how confidence intervals might be portrayed graphically. (SLD)