We describe best practices in reporting reliability estimates in counseling research with consideration to precision, generalization, and diverse populations. We provide a historical context to reporting reliability estimates, the limitations of past practices, and new methods to address reliability generalization. We highlight best practices…

This paper proposes the replication of Bird's (2010) and Serrano's (2011) studies, which examined distribution of practice effects in second language acquisition (SLA). These studies, which took place in authentic classroom contexts, produced conflicting results regarding the degree to which the learning of a second language (L2) benefited from…

In empirical applied linguistics research it is essential that the key variables are operationalized in a valid and reliable way, and that the scores are treated appropriately, allowing for a proper testing of the hypotheses under investigation. The current article addresses several theoretical and practical issues regarding the use of measurement…

Undergraduate research (UR) is a vetted educational tool that is commonly perceived to prepare students for entering graduate school and careers in STEM fields; however, scholarly attention to date has largely relied on self-report data, which may limit inferences about the causal effects on student outcomes. In light of this, recent calls have…

Describes how the assumptions underlying the use of multiple regression are not satisfied in reliability generalization studies and introduces mixed effects modeling to overcome many shortcomings of traditional approaches. Provides an example using results from the Beck Depression Inventory. (SLD)

The purposes of this paper are to identify common errors made by researchers when dealing with reliability coefficients and to outline best practices for reporting and interpreting reliability coefficients. Common errors that researchers make are: (1) stating that the instruments are reliable; (2) incorrectly interpreting correlation coefficients;…

Reviews issues, raised in part because of "Educational and Psychological Measurement" (EPM) policies, regarding "test reliability," which is psychometric terminology, and "score reliability," score-centric terminology. Discusses datametrics and provides a critique of T. Vacha-Haase's proposed meta-analytic reliability generalization via…

Responds to criticisms of some "Educational and Psychological Measurement" policies and the reliability generalization meta-analytic methods of T. Vacha-Haase. Explores consequences of misunderstanding score reliability. (SLD)

Discusses reliability issues in light of recent studies and debates focused on psychometrics versus datametrics terminology and reliabilities generalization. Discusses the way multiple perspectives on score reliability may affect research practice, editorial policies, and reliability generalization across studies. (SLD)

A reliability generalization study was completed on the Patterns of Adaptive Learning Survey achievement goal orientation scales to assess the prediction of (a) the different orientation scales, (b) the adaptation of items to meet research needs, (c) the number of respondents completing the instrument, and (d) the publication date cited for the…

Researchers consistently fail to report reliability estimates for data used in their studies. This lack of reporting hinders appropriate evaluation and interpretation of data and may lead to inappropriate conclusions. Because reliability is inured to scores obtained from a test, and not the test itself, it is important to report score reliability…

This paper notes the proliferation of literature about the characteristics of Japanese education and culture. Generalizations ranging from reports of remarkable progress in subject disciplines to praise of Japanese technology, social and individual behavior, and systematic accomplishments have been cited. It is contended in this study that some of…

This paper describes the planning of a qualitative research methodology to develop strategies to improve and expand social interaction skills of deaf-blind students in supported employment settings. Characteristics of qualitative research are delineated and include: definition of the domain of the research; an evolving study design; behavior…