Scoring high-dimensional assessments (e.g., > 15 traits) can be a challenging task. This paper introduces the multilabel neural network (MNN) as a scoring method for high-dimensional assessments. Additionally, it demonstrates how MNN can score the same test responses to maximize different performance metrics, such as accuracy, recall, or…

This paper reports an investigation into the psychometric properties of the COR-Advantage1.5 (COR-Adv1.5) assessment tool, a criterion-referenced observation-based instrument designed to assess the developmental abilities of children from birth through kindergarten. Using data from 8534 children participating in a state-funded preschool program…

Linking score scales across different tests is considered speculative and fraught, even at the aggregate level (Feuer et al., 1999; Thissen, 2007). We introduce and illustrate validation methods for aggregate linkages, using the challenge of linking U.S. school district average test scores across states as a motivating example. We show that…

This study investigates the relationships among factor correlations, inter-item correlations, and the reliability estimates of subscores, providing a guideline with respect to psychometric properties of useful subscores. In addition, it compares subscore estimation methods with respect to reliability and distinctness. The subscore estimation…

Assessing complex constructs such as those discussed under the umbrella of 21st century constructs highlights the need for a principled assessment design and validation approach. In our discussion, we made a case for three considerations: (a) taking construct complexity into account across various stages of assessment development such as the…

For decades, standardized reading comprehension tests have consisted of a series of passages and associated multiple-choice questions. Although widely used in and out of the classroom, there continues to be considerable disagreement regarding how or whether such tests have net value in the service of advancing educational progress in reading. This…

The purpose of this study was to examine the impact of dimensionality, common-item set format, and different scale linking methods on preserving equity property with mixed-format test equating. Item response theory (IRT) true-score equating (TSE) and IRT observed-score equating (OSE) methods were used under common-item nonequivalent groups design.…

"Linking" is a term given to a general class of procedures by which one represents scores X on one test or measure in terms of scores Y on another test or measure. A recent taxonomy by Holland and Dorans (2006; Holland, 2007) organizes the various types of links into three broad categories: prediction, scale aligning, and equating. In…

Used aggregate ratings of teacher behavior as data for a multitrait-multimethod validity analysis. Scaled ratings using Rasch latent trait scaling model and traditional scaling techniques. Compared Rasch-scaled multitrait-multimethod matrix to the traditionally scaled multitrait-multimethod matrix. Results showed Rasch scaling resulted in higher…

This study used a sample (n=967) of academically gifted adolescent students attending summer enrichment programs and participating in urban school districts' gifted programs to evaluate the reliability and validity of the Adolescent Coping Scale. Results suggest the instrument is sufficiently reliable for group administration and research purposes…

During the past 6 years, under a contract from the National Center for Education Statistics, a Technical Review Panel has overseen and conducted a series of research studies addressing a range of validity questions relevant to the various uses and interpretations of the National Assessment of Educational Progress (NAEP). Study topics included: (1)…

Increased demands for colleges and universities to engage in outcomes assessment for accountability purposes have accelerated the need to bridge the gap between higher education practice and the fields of measurement, assessment, and evaluation. The "Handbook on Measurement, Assessment, and Evaluation in Higher Education" provides higher…

In the early twentieth century behaviorist Edward L. Thorndike began the development and use of measurement scales to replace personal judgment to evaluate student compositions in U.S. public schools. In 1912, utilizing the Fullerton and Catell equal difference theorem, Milo B. Hillegas released the first scientifically designed scale to measure…

The applicability of a recognized construct validation procedure, the nomological network, to a sixth-grade criterion-referenced mathematics test developed in a metropolitan school system was investigated. Two samples were tested. The first sample was used to investigate two hypotheses: items within each objective will constitute a Guttman scale,…

This study compared college students' responses on their evaluations of the effectiveness of full- and part-time college faculty. A group of 1,101 students completed evaluation instruments for all courses taught by full-time faculty, and 2,067 students completed evaluations for all courses taught by part-time faculty in spring 1998. In fall 1998,…