When cognitive and educational tests are administered under time limits, tests may become speeded and this may affect the reliability and validity of the resulting test scores. Prior research has shown that time limits may create or enlarge gender gaps in cognitive and academic testing. On average, women complete fewer items than men when a test…

This article extends the procedure outlined in the article by Raykov, Marcoulides, and Tong for testing congruence of latent constructs to the setting of binary items and clustering effects. In this widely used setting in contemporary educational and psychological research, the method can be used to examine if two or more homogeneous…

The study aimed to develop the Homophobic Bullying Scale and to investigate its psychometric properties. The items of the Homophobic Bullying Scale were created to measure high school students' bullying behaviors motivated by homophobia, including verbal bullying, relational bullying, physical bullying, property bullying, sexual harassment, and…

Self-report surveys are widely used to measure adolescent risk behavior and academic adjustment, with results having an impact on national policy, assessment of school quality, and evaluation of school interventions. However, data obtained from self-reports can be distorted when adolescents intentionally provide inaccurate or careless responses.…

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental…

This article is concerned with developing a measure of general academic ability (GAA) for high school graduates who apply to colleges, as well as with the identification of optimal weights of the GAA indicators in a linear combination that yields a composite score with maximal reliability and maximal predictive validity, employing the framework of…

The current study evaluated the relationship between various operationalizations of the Advanced Placement[R] (AP) exam and course information with first-year grade point average (FYGPA) in college to better understand the role of AP in college admission decisions. In particular, the incremental validity of the different AP variables, above…

The many-faceted Rasch (MFR) model has been used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR model for examining the quality of ratings obtained from a standard…

Statistical biases associated with single-level analyses underscore the importance of partitioning variance/covariance matrices into individual and group levels. From a multilevel perspective based on data from 21,579 students in 58 high schools, the present study assesses the multilevel factor structure of motivation and engagement with a…

The purpose of this validity study was to improve measurement of athletes' evaluations of their head coach's coaching competency, an important multidimensional construct in models of coaching effectiveness. A revised version of the Coaching Competency Scale (CCS) was developed for athletes of high school teams (APCCS II-HST). Data were collected…

The purpose of this study is to test the validity of scores on the Homework Purpose Scale using 681 rural and 306 urban high school students. First, confirmatory factor analysis was conducted on the rural sample. The results reveal that the Homework Purpose Scale comprises three separate yet related factors, including Learning-Oriented Reasons,…

This study investigates the effect of mode of administration of the Raven Standard Progressive Matrices test on distribution, accuracy, and meaning of raw scores. A random sample of high school students take counterbalanced paper-and-pencil and computer-based administrations of the test and answer a questionnaire surveying preferences for…

This article presents a theoretical rationale and empirical evidence regarding the validity of scores obtained from two competing approaches to operationalizing scale items to measure emergent organizational properties. The authors consider whether items in scales intended to measure organizational properties should prompt survey takers to provide…

The purpose of this validity study was to improve measurement of coaching efficacy, an important variable in models of coaching effectiveness. A revised version of the coaching efficacy scale (CES) was developed for head coaches of high school teams (CES II-HST). Data were collected from head coaches of 14 relevant high school sports (N = 799).…

The current study compared student performance between paper-and-pencil testing (PPT) and computer-based testing (CBT) on a large-scale statewide end-of-course English examination. Analyses were conducted at both the item and test levels. The overall results suggest that scores obtained from PPT and CBT were comparable. However, at the content…