Item response tree (IRTree) models are a flexible framework to control self-reported trait measurements for response styles. To this end, IRTree models decompose the responses to rating items into sub-decisions, which are assumed to be made on the basis of either the trait being measured or a response style, whereby the effects of such person…

Scholastic tests regard cognitive abilities to be domain-specific competences. However, high correlations between competences indicate either high task similarity or a dependence on common factors. The present rating study examined the validity of 12 Programme for International Student Assessment (PISA) and Third or Trends in International…

In adolescence research, the treatment of measurement reliability is often fragmented, and it is not always clear how different reliability coefficients are related. We show that generalizability theory (G-theory) is a comprehensive framework of measurement reliability, encompassing all other reliability methods (e.g., Pearson "r,"…

The purpose of this report is to review ETS psychometric contributions that focus on test scores. Two major sections review contributions based on assessing test scores' measurement characteristics and other contributions about using test scores as predictors in correlational and regression relationships. An additional section reviews additional…

In recent years, the social sciences have seen a resurgence in the study of divergent thinking (DT) measures. However, many of these recent advances have focused on abstract, decontextualized DT tasks (e.g., list as many things as you can think of that have wheels). This study provides a new perspective by exploring the reliability and validity…

In this study we use data from the Early Childhood Longitudinal Survey third- and fifth-grade samples to investigate teacher judgments of student achievement, the extent to which they offer a similar picture of student mathematics achievement compared to standardized test scores, and whether classroom assessment practices moderate the relationship…

A large number of seemingly diverse coefficients have been proposed as indices of dependability, or reliability, for domain-referenced and/or mastery tests. In this paper, it is shown that most of these indices are special cases of two generalized indices of agreement: one that is corrected for chance, and one that is not. The special cases of…

This study is an attempt at a cohesive characterization of the concept of essay reliability. As such, it takes as a basic premise that previous and current practices in reporting reliability estimates for essay tests have certain shortcomings. The study provides an analysis of these shortcomings--partly to encourage a fuller understanding of the…

Some aspects of test reliability are discussed. Topics covered are: (1) how high should a reliability coefficient be?; (2) two factors affecting the interpretation of reliability coefficients--range of talent and interval between testings; (3) some common misconceptions--reliability of speed tests, part vs. total reliability, reliability for what…

Ordinal measurement is the rank ordering of individuals in a population. For ordinal measurement, the concept of an individual propensity distribution is his or her true score. Estimation of, as well as other aspects of the distribution, are discussed. (Author/JKS)

A procedure for estimating the reliability of a factorially complex composite is considered. An application of its use with Scholastic Aptitude Test data is provided. (Author/JKS)

Samples in research on individual and group differences may be selected based on whole scores which differ from the population mean. Children are diagnosed in clinical practice with a whole score. These procedures produce regression to the population mean which can affect accuracy and adequacy of part score interpretations. (Author/DWH)

A computer program providing comprehensive test and item analysis is presented. Completing its performance on one run, the program, written in Fortran and emphasizing ease of use, integrates various statistical techniques for analyzing individual items and the overall test, in addition to generating a variety of standard scores. (Author/JKS)

Comparison of the Goodenough-Harris and McCarthy scoring procedures for 60 kindergarten children's drawings yielded substantial agreement between the two scoring systems. The streamlined McCarthy scoring system should be utilized when large numbers of children are being evaluated with short periods of time. (Author)