Generalizability theory is a modern and powerful framework for conducting reliability analyses. It is flexible to accommodate both random and fixed facets. However, there has been a relative scarcity in the practical literature on how to handle the fixed facet case. This article aims to provide practitioners a conceptual understanding and…

Background/Context: Over the past twenty plus years, the What Works Clearinghouse (WWC) has reviewed over 1,700 studies, cataloging effect sizes for 189 interventions. Some 56% of these interventions include results from multiple, independent studies; on average, these include results of [approximately]3 studies, though some include as many as 32…

This study presents several multivariate Generalizability theory designs for analyzing automatic item-generated (AIG) based test forms. The study used real data to illustrate the analysis procedure and discuss practical considerations. We collected the data from two groups of students, each group receiving a different form generated by AIG. A…

This paper examines whether general intelligence (g) factors derived from different test batteries are equivalent. There are three views regarding the equivalency of g-factors: (1) "indicator indifference" claims that test content is irrelevant as long as g loadings are identical and that single tests can be adequate indicators of g; (2)…

We demonstrate how to analyze complete multivariate generalizability theory (GT) designs within structural equation modeling frameworks that encompass both individual subscale scores and composites formed from those scores. Results from numerous analyses of observed scores obtained from respondents who completed the recently updated form of the…

Given that the contextual interference (CI) phenomenon is one of the most extensively studied and debated topics in motor learning--featured prominently in scientific literature, textbooks, and practitioner guides--it is unsurprising that recent meta-analyses on the topic have generated critical discussion and contrasting interpretations. This…

Perception and perceptual memory play crucial roles in fear generalization, yet their dynamic interaction remains understudied. This research (N = 80) explored their relationship through a classical differential conditioning experiment. Results revealed that while fear context perception fluctuates over time with a drift effect, perceptual memory…

In this article, we pursue two points of discussion. First, a new illustration is presented of the person-oriented tenet according to which it can be hazardous to generalize to the individual results that are based on the analysis of aggregated data. Second, it is illustrated that taking into account serial dependence information can result in not…

Learning analytics is a fast-growing discipline. Institutions and countries alike are racing to harness the power of using data to support students, teachers and stakeholders. Research in the field has proven that predicting and supporting underachieving students is worthwhile. Nonetheless, challenges remain unresolved, for example, lack of…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

Purpose: A considerable number of screening and diagnostic tools for autism exist, but variability in these measures presents challenges to data harmonization and the comparability and generalizability of findings. At the same time, there is a movement away from autism symptomatology to stances that capture heterogeneity and appreciate diversity.…

Value added models (VAMs) attempt to estimate the causal effects of teachers and schools on student test scores. We apply Generalizability Theory to show how estimated VA effects depend upon the selection of test items. Standard VAMs estimate causal effects on the items that are included on the test. Generalizability demands consideration of how…

This study addresses the role of awareness in learning and the variables that may facilitate adult second language (L2) implicit learning. We replicated Williams's (2005) study with a similar group of academic learners enrolled at university as well as a group of non-college-educated adults in order to explore the generalizability of the findings…

Computing confidence intervals around generalizability coefficients has long been a challenging task in generalizability theory. This is a serious practical problem because generalizability coefficients are often computed from designs where some facets have small sample sizes, and researchers have little guide regarding the trustworthiness of the…

Although generalizability theory (GT) designs typically are analyzed using analysis of variance (ANOVA) procedures, they also can be integrated into structural equation models (SEMs). In this tutorial, we review basic concepts for conducting univariate and multivariate GT analyses and demonstrate advantages of doing such analyses within SEM…