A hypothetical case study provides examples of the inter-rater reliability issues involved in complex performance assessment, focusing on the Baldrige model. A hypothetical team of five evaluators was asked to rate a Baldrige model performance assessment along the seven defined criteria or performance dimensions that comprise the Baldrige model…

This report describes a reliability and validity study on a learning styles instrument that was developed based on the Dunn, Dunn, & Price model. That model included 104 Likert five-point scale items for investigating 24 scales grouped into five categories considered likely to affect learning. The Learning Style Preference Inventory (LSPI)…

This paper explores how facets of the concept "rigor" might be applied to questions about the validity and reliability of research independently of the research modes. The focus of the critical lens could then be on how to assess the contribution of various forms of research rather than on the "paradigm wars" and arguments…

The purpose of this paper is to highlight some psychometric cautions that should be observed when seeking to develop short form versions of tests. Several points are made: (1) score reliability is impacted directly by the characteristics of the sample and testing conditions; (2) sampling error has a direct influence on reliability and factor…

The present study developed and assessed reliability and validity of a new instrument, the Intercultural Sensitivity Scale (ISS). Based on a review of the literature, 44 items thought to be important for intercultural sensitivity were generated. A sample of 414 college students rated these items and generated a 24-item final version of the…

This study examined the reliability of scores assigned to the essays written by Kentucky students to meet the University Writing Requirement (UWR) at Eastern Kentucky University. Two sets of essays, 50 each, on the same prompt that had been read and scored in 1989 and 1997 by trained UWR scorers were read by 7 UWR scorers in 2000. A correlation…

With the rise of violent teenage crime, with an alarming number of child soldiers across the globe, and with the continually increasing number of children and adolescents who are victimized by violence and war, an instrument that measures nonviolent tendencies would be very useful. The Teenage Nonviolence Test (TNT) was recently developed and…

This research study was designed to establish the reliability of the Teenage Nonviolence Test (TNT). The consistency and factor structure of the TNT using a sample of 376 adolescents were evaluated. The stability of the TNT was assessed over time by administering the TNT twice with a two week intervening interval to 87 adolescents. The TNT appears…

The Learning Style Inventory (LSI; Kolb, 1976; 1985 ) is a commonly used measure of learning styles based on Kolbs Experiential Learning Model. The psychometric soundness of LSI scores has been critiqued historically. This study reviewed the literature on the LSI and evaluated the psychometric properties of Kolbs original and revised versions of…

This Critical Issue Bibliography (CRIB) Sheet presents resources on college rankings publications, criticisms of rankings methodology, effects of the rankings on the public, and alternatives to the major rankings guides. The annotated bibliography lists 5 Internet resources and 17 other resources, all of which are in the ERIC database. (SLD)

A number of states have school building accountability systems that rely on comparisons of achievement from one year to the next. Improvement of the performance of schools is judged by changes in the achievement of successive groups of students. Year-to-year changes in scores for successive groups of students have a great deal of volatility. The…

Previous studies have indicated that the reliability of test scores composed of testlets is overestimated by conventional item-based reliability estimation methods (S. Sireci, D. Thissen, and H. Wainer, 1991; H. Wainer, 1995; H. Wainer and D. Thissen, 1996; G. Lee and D. Frisbie). In light of these studies, it seems reasonable to ask whether the…

The primary purpose of this study was to investigate the appropriateness and implication of incorporating a testlet definition into the estimation of the conditional standard error of measurement (SEM) for tests composed of testlets. The five conditional SEM estimation methods used in this study were classified into two categories: item-based and…

Teacher efficacy has proven to be an important variable in teacher effectiveness. It is consistently related to positive teaching behaviors and student outcomes. However, the measurement of this construct is the subject of current debate, which includes critical examination of predominant instruments used to assess teacher efficacy. The present…

Classical test theory is based on the concept of a true score for each examinee, defined as the expected or average score across an infinite number of repeated parallel tests. In most cases, there is only a score from a single administration of the test in question. The difference between this single observed score and the underlying true score is…