Over the past ten years, propensity score methods have made an important contribution to improving generalizations from studies that do not select samples randomly from a population of inference. However, these methods require assumptions and recent work has considered the role of bounding approaches that provide a range of treatment impact…

This article serves as an introduction to the Special Issue on "Decolonizing and Diversifying Research in Cognitive Development." The Special Issue comprises six articles: two articles are empirical articles that focus on executive function development in under-represented environments, two articles address barriers pathways toward…

In this study, generalizability theory was used to examine the extent to which (a) time-sampling methodology, (b) number of simultaneous behavior targets, and (c) individual raters influenced variance in ratings of academic engagement for an elementary-aged student. Ten graduate-student raters, with an average of 7.20 hr of previous training in…

The Angoff method requires experts to view every item on the test and make a probability judgment. This can be time consuming when there are large numbers of items on the test. In this study, a G-theory framework was used to determine if a subset of items can be used to make generalizable cut-score recommendations. Angoff ratings (i.e.,…

To understand the extent to which the general-factor loadings of tests are inherent in their characteristics or due to the sampling of tests, the number of tests in the correlation matrix, and the factor-extraction methods used to obtain them, test scores from a large sample of young adults were inserted into independent and overlapping batteries…

Background: Educational researchers have long espoused the virtues of writing with regard to student cognitive skills. However, research on the reliability of the grades assigned to written papers reveals a high degree of contradiction, with some researchers concluding that the grades assigned are very reliable whereas others suggesting that they…

The problem of predicting universe scores for samples of examinees based on their responses to samples of items is treated. The measurement model categorizes items according to the cells of a table of test specifications, and the linear function derived for minimizing error variance in prediction uses responses to these categories. (Author/JKS)

Reviews the criticisms of sampling assumptions in generalizability theory (and in reliability theory) and examines the feasibility of using representative sampling, stratification, homogeneity assumptions, and replications to address these criticisms. Suggests some general outlines for the conduct of generalizability theory studies. (SLD)

Studied three procedures for estimating the standard errors of school passing rates using a generalizability theory model and considered the effects of student sample size. Results show that procedures differ in terms of assumptions about the populations from which students were sampled, and student sample size was found to have a large effect on…

A structural equation modeling approach to scale reliability evaluation can be employed to estimate generalizability theory indexes in settings where sampling of subjects and conditions is carried out. In one- and two-facet crossed designs, it is demonstrated how this method can be used to obtain estimates of relative generalizability…

In this article, we critically evaluate the expectations of assistant professors as perceived by junior and senior faculty (n=22) across seven disciplines at two universities. Our research shows that there are similarities and differences within Sociology departments, across disciplines, and across the teaching and research schools where we…

Studied the sampling variability of estimated variance components using data collected over several years for a listening and writing performance assessment and evaluated the stability of estimated measurement precision. Results indicate that the estimated variance components varied from one year to another and suggest that the measurement…

Khuri's and Satterthwaite's methods of obtaining confidence intervals of variance components are compared. The article discusses that Khuri's method may be applied to obtain confidence intervals for the variance components and other linear functions of the expected mean squares used in generalizability theory. (Author/JAZ)

Presents solutions for the problem of maximizing the generalizability coefficient under a budget constraint. Shows that the Cauchy-Schwarz inequality can be applied to derive optimal continuous solutions for the number of conditions of each facet. Illustrates the formal similarity between optimization problems in survey sampling and…

Description of common errors found in the statistical methodologies of research carried out by librarians, focuses on sampling and generalizability. The discussion covers the need to either adapt library research to the statistical abilities of librarians or to educate librarians in the proper use of statistics. (15 references) (CLB)