The dimensional comparison theory (DCT) focuses on the effects of internal, dimensional comparisons (e.g., "How good am I in math compared to English?") on academic self-concepts with widespread consequences for students' self-evaluation, motivation, and behavioral choices. DCT is based on the internal/external frame of reference model…

The present paper aims to discuss how data from international large-scale assessments (ILSAs) can be utilized and combined, even with other existing data sources, in order to monitor educational outcomes and study the effectiveness of educational systems. We consider different purposes of linking data, namely, extending outcomes measures,…

The article discusses comparative organizational studies, focusing on change in higher education institutions, and in particular (but not only) on studies that compare organizations in different national settings. It first presents and discusses different approaches to comparison based on a couple of typologies that have been developed to identify…

Previous assessments of the reliability of test scores for testlet-composed tests have indicated that item-based estimation methods overestimate reliability. This study was designed to address issues related to the extent to which item-based estimation methods overestimate the reliability of test scores composed of testlets and to compare several…

Categorization is one of the fundamental building blocks of cognition, and the study of categorization is notable for the extent to which formal modeling has been a central and influential component of research. However, the field has seen a proliferation of noncomplementary models with little consensus on the relative adequacy of these accounts.…

The purpose of this study was to investigate the methods of estimating the reliability of school-level scores using generalizability theory and multilevel models. Two approaches, "student within schools" and "students within schools and subject areas," were conceptualized and implemented in this study. Four methods resulting from the combination…

I define a treatment effect in terms of a comparison of outcomes and provide a typology of all possible comparisons that can be used to estimate treatment effects, including comparisons that are relatively unknown in both the literature and practice. I then assess the relative merit, worth, and value of all possible comparisons based on the…

Self- and peer assessments are becoming more popular in classrooms, but there are few data on the reliability and validity of such assessments performed by school children. Because these factors are greatly affected by the number of raters, we conducted two studies to determine the rating behaviours of teenagers in self- and peer assessments, and…

Generalizability theory provides a technique for accurately estimating the reliability of measurements. The power of this theory is based on the simultaneous analysis of multiple sources of error variances. Equally important, generalizability theory considers relationships among the sources of measurement error. Just as multivariate inferential…

In the first part of this paper we discuss the feasibility of using Generalizability (G) Theory to examine the dependability of concept map assessments and to design a concept map assessment for a particular practical application. In the second part, we apply G theory to compare the technical qualities of two frequently used mapping techniques:…

Studied the appropriateness and implications of incorporating a testlet definition into the estimation of procedures of the conditional standard error of measurement (SEM) for tests composed of testlets. Simulation results for several methods show that an item-based method using a generalizability theory model provided good estimates of the…

Three procedures for evaluating the sampling specificity of results are reviewed. These procedures are Tukey's jacknife technique, Efron's bootstrap technique, and cross-validation methods. The jacknife technique uses different subsamples derived from the original total data set to provide empirical estimates of the generalizability of effect…

A 2-facet measurement model is identified, and its coefficient of generalizability (CG) is examined. Three other multifaceted measurement models and their CGs are identified. An empirical investigation of all four procedures is conducted using data from a study of the psychopathology of 71 prison inmates. (SLD)

Invariance procedures have been known for over three decades. Through the years, these procedures have not been widely used by researchers. One reason for the lack of use is that most of the articles on invariance procedures have been mathematically oriented. This mathematical orientation and the lack of mathematical background on the part of most…

Not infrequently, investigators assume that reliability for groups is greater than reliability for persons, or that the error variance for groups is less than that for persons. Using generalizability theory, it is shown that this "conventional wisdom" is not necessarily true. Examples are provided from the course-evaluation and the…