Although generalizability theory (GT) designs have traditionally been analyzed within an ANOVA framework, identical results can be obtained with structural equation models (SEMs) but extended to represent multiple sources of both systematic and measurement error variance, include estimation methods less likely to produce negative variance…

Though the Chemistry Self-Concept Inventory (CSCI) was developed to study one aspect of the affective domain in college chemistry students, the instrument on which it was based, the Self-Description Questionnaire III, was developed for use with late adolescents. As such, we explored data generated from administering the CSCI to high school…

This article presents a comparative judgment approach for holistically scored constructed response tasks. In this approach, the grader rank orders (rather than rate) the quality of a small set of responses. A prior automated evaluation of responses guides both set formation and scaling of rankings. Sets are formed to have similar prior scores and…

Two studies were conducted for the development and validation of a multidimensional test to assess undergraduate students' mathematical thinking about derivative. The first study involved two phases: question generation and refinement of the Thinking-about-Derivative Test (TDT). The second study included four phases as follows: test…

Component Universe Score Profile analysis (CUSP) is introduced in this paper as a psychometric alternative to multivariate profile analysis. The theoretical foundations of CUSP analysis are reviewed, which include multivariate generalizability theory and constrained principal components analysis. Because CUSP is a combination of generalizability…

Background: Interest in the role of metacognition has been steadily rising in most forms of education. This study focuses on the construction of a questionnaire for measuring metacognitive knowledge, metacognitive regulation and metacognitive responsiveness among students in higher education and the subsequent process of testing to determine its…

Floyd, Shands, Rafael, Bergeron and McGrew (2009) used generalizability theory to test the reliability of general-factor loadings and to compare three different sources of error in them: the test battery size, the test battery composition, the factor-extraction technique, and their interactions. They found that their general-factor loadings were…

Three essay questions were administered to junior education major college students. Factor analysis of the ratings showed that content played a large role in the students' responses, yielding dominant first order factors. Generalizability theory, used to examine the reliability of students' ratings, showed the need for more raters and questions.…

Effects of different weighting schemes on selecting the optimal number of observations in multivariate-multifacet generalizability designs are studied when cost constraints are imposed. Comparison of four schemes through simulation indicates that all four produce similar optimal values and that reliability should be similar. (SLD)

The extent to which a scale score generalizes to a latent variable common to all of the scale's indicators is indexed by the scale's general factor saturation. Seven techniques for estimating this parameter--omega[hierarchical] (omega[subscript h])--are compared in a series of simulated data sets. Primary comparisons were based on 160 artificial…