Conceptual replications have received increased coverage in the educational research agenda. This article argues for clarity in, and justification of, the definition, scope, and boundaries of a conceptual replication and what it can and cannot do. It argues for clear justifications when changing components from those of the original study. The…

Special education researchers conduct studies that can be considered replications. However, they do not often refer to them as replication studies. The purpose of this article is to consider the potential benefits of conceptualizing special education intervention research within a framework of systematic, conceptual replication. Specifically, we…

In recent years, the regression discontinuity design (RDD) has gained widespread recognition as a quasi-experimental method that when used correctly, can produce internally valid estimates of causal effects of a treatment, a program or an intervention (hereafter referred to as treatment effects). In an RDD study, subjects or groups of subjects…

I define a treatment effect in terms of a comparison of outcomes and provide a typology of all possible comparisons that can be used to estimate treatment effects, including comparisons that are relatively unknown in both the literature and practice. I then assess the relative merit, worth, and value of all possible comparisons based on the…

Standard-setting methods are widely used to determine cut scores on a test that examinees must meet for a certain performance standard. Because standard setting is a measurement procedure, it is important to evaluate variability of cut scores resulting from the standard-setting process. Generalizability theory is used in this study to estimate…

lllustrates a link between the multiple-rater kappa of J. Fleiss (1971) or other analogues and the generalizability (G) coefficient for a single facet design, and discusses the use and interpretation of G theory in the study of interrater agreement when data are measured on a nominal scale. (SLD)

Simulations demonstrate the effects of correlated errors on the person-by-occasion design in which the confounding effect of equal time intervals results in correlated error terms in the linear model. Two specific error correlation structures were examined. Conditions under which underestimation and overestimation occur are discussed. (SLD)

A methodology for determining the optimal number of observations to use in a measurement design when resource constraints are imposed is presented. Two- and three-facet designs are outlined. Parallel closed form formulae can easily be determined for other designs. (TJH)

An efficient search procedure is presented for determining the optimal number of observations of facets in a design that maximize generalizability when resource constraints are imposed. The procedure is illustrated for three-facet and four-facet designs, with extensions for other configurations. (Author/SLD)

Replication is important to viable scientific inquiry; results that will not replicate or generalize are of very limited value. Statistical significance enables the researcher to reject or not reject the null hypothesis according to the sample results obtained, but statistical significance does not indicate the probability that results will be…

A study on sampling errors of variance components was conducted within the framework of generalizability theory by P. L. Smith (1978). The study used an intuitive approach for solving the problem of how to allocate the number of conditions to different facets in order to produce the most stable estimate of the universe score variance. Optimization…

This article argues that the 1981 work of Carbno presented unwarranted conclusions because its design included an improper operationalization of the object of measurement, given the problems addressed, and because the sample sizes employed were too small. (Author/PN)

This paper presents a review of the literature on reliability in qualitative studies. Reliability is defined as the extent to which studies can be replicated, using the same methods, and getting the same results. It is the degree to which data are independent of the accidental circumstances of the research. The review includes the following three…

A method is presented for determining the optimal number of conditions to use in multivariate-multifacet generalizability designs when resource constraints are imposed. A decision maker can determine the number of observations needed to obtain the largest possible generalizability coefficient. The procedure easily applies to the univariate case.…

A methodology is presented for minimizing mean error variance in generalizability studies when resource constraints are imposed. The optimal number of observations and conditions of facets for random model, fully crossed one- and two-facet designs can be decided. Parallel closed form formulas can be determined for other designs. (SLD)