We describe best practices in reporting reliability estimates in counseling research with consideration to precision, generalization, and diverse populations. We provide a historical context to reporting reliability estimates, the limitations of past practices, and new methods to address reliability generalization. We highlight best practices…

Experimental psychology is witnessing an increase in research on individual differences, which requires the development of new tasks that can reliably assess variations among participants. To do this, cognitive researchers need statistical methods that many researchers have not learned during their training. The lack of expertise can pose…

The historical turn in social science has prompted scholars to engage with the work of historians on a large scale. Here, social scientists face two standard problems of selection bias: confirmation bias and convenience sampling. So far, the record of dealing with these problems has been poor, and little has been done to specify how social…

Researchers and policymakers in the US and beyond increasingly seek to identify teaching qualities that are associated with academic achievement gains for K-12 students through effectiveness studies. Yet teaching quality varies with academic content and social contexts, involves multiple participants, and requires a range of skills, knowledge, and…

This brief paper introduces a new approach to assessing the trustworthiness of research comparisons when expressed numerically. The 'number needed to disturb' a research finding would be the number of counterfactual values that can be added to the smallest arm of any comparison before the difference or 'effect' size disappears, minus the number of…

A common question asked by researchers using regression models is, What sample size is needed for my study? While there are formulae to estimate sample sizes, their assumptions are often not met in the collected data. A more realistic approach to sample size determination requires more information such as the model of interest, strength of the…

For testing programs that administer multiple forms within a year and across years, score equating is used to ensure that scores can be used interchangeably. In an ideal world, samples sizes are large and representative of populations that hardly change over time, and very reliable alternate test forms are built with nearly identical psychometric…

This discussion constructs a survey data quality strategy for institutional researchers in higher education in light of total survey error theory. It starts with describing the characteristics of institutional research and identifying the gaps in literature regarding survey data quality issues in institutional research and then introduces the…

This descriptive study sought to determine the attitudes of middle school teachers in Indiana, Kentucky, and Ohio toward the current implementation of key components of the middle school concept within their schools. Researchers were asked to assess the relationship of state accountability programs to the implementation levels of such components.…

For a test that consists of dichotomously scored items, several approaches have been reported in the literature for estimating classification consistency and accuracy indices based on a single administration of a test. Classification consistency and accuracy have not been studied much, however, for "complex" assessments--for example,…

Progress monitoring is an important component of effective instructional practice. Curriculum-based measurement (CBM) is a form of progress monitoring that has been the focus of rigorous research. Two approaches for formulating CBM systems exist. The first is to assess performance regularly on a task that serves as a global indicator of competence…

This study explores a strategy for improving the stability of variance component estimates when only small samples are available, using a series of small, less complex generalizability (G) study designs as a surrogate for a single large design. (Author/BW)

Reviews the criticisms of sampling assumptions in generalizability theory (and in reliability theory) and examines the feasibility of using representative sampling, stratification, homogeneity assumptions, and replications to address these criticisms. Suggests some general outlines for the conduct of generalizability theory studies. (SLD)

Discusses sampling bias problems in the use of the greatest lower bound (g.l.b.) to reliability and offers explicit expressions for the second order derivatives. This yields closed form expression for the asymptotic bias of both the g.l.b. and its numerator. Illustrates the approach through a numeric example. (SLD)

Illustrates that sampling variance can be very large when a three-wave quasi simplex model is used to obtain reliability estimates. Also shows that, for the reliability parameter to be identified, the model assumes a Markov process. These problems are evaluated with both real and Monte Carlo data. (SLD)