The reliability of a test score is usually underestimated and the deflation may be profound, 0.40 - 0.60 units of reliability or 46 - 71%. Eight root sources of the deflation are discussed and quantified by a simulation with 1,440 real-world datasets: (1) errors in the measurement modelling, (2) inefficiency in the estimator of reliability within…

Recent years have seen a growing number of studies investigating the accuracy of nonprobability online panels; however, response quality in nonprobability online panels has not yet received much attention. To fill this gap, we investigate response quality in a comprehensive study of seven nonprobability online panels and three probability-based…

As part of an ongoing investigation of students' learning in first semester upper-division quantum mechanics, we needed a high-quality conceptual assessment instrument for comparing outcomes of different curricular approaches. The process of developing such a tool started with converting a preliminary version of a 14-item open-ended quantum…

A scale was developed to measure perceptions of sexual abuse in youth sports by assessing (a) the perceived prevalence of sexual abuse committed by pedophilic youth sport coaches, (b) the perceived likelihood that a coach is a pedophile, (c) perceptions on how youth sport organizations should manage the risk of pedophilia, and (d) media influence…

The aim of this study was to adapt and evaluate the Developmental Coordination Disorder Questionnaire (DCDQ) for use in Chinese-speaking countries. A total of 1082 parents completed the DCDQ and 35 parents repeated it after 2 weeks for test-retest reliability. Two items were deleted after examination of test consistency. Cronbach's [alpha] for the…

Using Rasch analysis, the psychometric properties of a newly developed 35-item parent-proxy instrument, the Caregiver Assessment of Movement Participation (CAMP), designed to measure movement participation problems in children with Developmental Coordination Disorder, were examined. The CAMP was administered to 465 school children aged 5-10 years.…

Bilevel dimensionality of probability was examined via factor analysis, Rasch latent trait analysis, and classical item analysis. Results suggest that when nonstandardized measures are the criteria for achievement, relying solely on estimates of content validity may lead to erroneous interpretation of test score data. (JKS)

Procedures for computing content validity and consistency reliability coefficients and determining the statistical significance of these coefficients are described. Procedures employing the multinomial probability distribution for small samples and normal curve probability estimates for large samples, can be used where judgments are made on…

A latent trait method is presented to investigate the possibility that Angoff or Nedelsky judges specify inconsistent probabilities in standard setting techniques for objectives-based instructional programs. It is suggested that judges frequently specify a low probability of success for an easy item but a large probability for a hard item. The…