Quantitative content analysis is increasingly used to surpass surface level analyses in computer-supported collaborative learning (e.g., counting messages), but critical reflection on accepted practice has generally not been reported. A review of CSCL conference proceedings revealed a general vagueness in definitions of units of analysis. In…

Ratings of child involvement in manual-based cognitive-behavioral treatment for anxiety were associated with the absence of primary anxiety diagnosis and reductions in impairment ratings at posttreatment for 59 children with anxiety (ages 8-14 years). Good-to-excellent interrater reliability was established for the independent ratings of 237…

The reliability and validity of the Attachment Q Sort (AQS; Waters & Deane, 1985) was tested in a series of meta-analyses on 139 studies with 13,835 children. The observer AQS security score showed convergent validity with Strange Situation procedure (SSP) security (r=31) and excellent predictive validity with sensitivity measures (r=39). Its…

Reliability is a problem inherent in all educational assessments, but the amount of attention this particular problem should be given is related to the function and use of the assessment. In this article, classification accuracy is put forward as a conceptualization of reliability that is meaningful for a large number of educational assessments.…

The information function is an important statistic in item response theory (IRT) applications. Although the information function is often described as the IRT version of reliability, it differs from the classical notion of reliability from a critical perspective: replication. This article first explores the information function for the…

The authors developed and validated a measure of employees' attitudes toward lateness at work. Analyses provided clear evidence of the reliability and validity of the new measure. Specifically, high reliabilities were observed in both student (a = 0.82) and employee (a = 0.84) samples. Using objective lateness data from organizations, the measure…

The Childhood Asperger Syndrome Test (CAST) is a 37-item parental self-completion questionnaire to screen for autism spectrum conditions in research. Good test accuracy was demonstrated in studies with primary school aged children in mainstream schools. The aim of this study was to investigate the test-retest reliability of the CAST. Parents of…

A new test index is defined as the probability of obtaining two randomly selected test scores (PDTS) as statistically different. After giving a concept definition of the test index, two simulation studies are presented. The first analyzes the influence of the distribution of test scores, test reliability, and sample size on PDTS within classical…

Uses and consequences of educational testing have increased dramatically in recent years. Professional standards to ensure fair treatment of all affected by test results are more important than ever, but standards for developing and using educational tests are only helpful if they are followed. Test developers and users each have a role to play in…

The measurement of online self-regulation processes is a very important issue and in this rejoinder to Ainley and Patrick (this issue) I am arguing that including measures of metacognitive experiences, in conjunction with measures of other affective experiences, in various phases of task processing can increase the reliability and validity of…

This article presents a formula for weighted kappa in terms of rater means, rater variances, and the rater covariance that is particularly helpful in emphasizing that weighted kappa is an absolute agreement measure in the sense that it is sensitive to differences in rater's marginal distributions. Specifically, rater mean differences will decrease…

The CAGE is a commonly used alcohol screening instrument. Although considerable work has been done on the validity of CAGE scores, relatively little information is available on their reliability. Reliability induction and generalization studies were performed for the CAGE. Of the 259 studies available for analysis, only 19 (7.3%) contained…

Differential item functioning (DIF) analyses are used to identify items that operate differently between two groups, after controlling for ability. The Simultaneous Item Bias Test (SIBTEST) is a popular DIF detection method that matches examinees on a true score estimate of ability. However in some testing situations, like test translation and…

The revised push-up test has been found to have good validity but it produces many zero scores for women. Maybe there should be an alternative to the revised push-up test for college-age women. The purpose of this study was to determine the objectivity, reliability, and validity for the bent-knee push-up test (executed on hands and knees) for…

Purpose: This article critically reviews evidence to determine whether the use of disfluency typologies, such as "syllable repetitions" or "prolongations", has assisted the understanding or treatment of developmental stuttering. Consideration is given to whether there is a need for a fundamental shift in the basis for constructing measures of…