We describe best practices in reporting reliability estimates in counseling research with consideration to precision, generalization, and diverse populations. We provide a historical context to reporting reliability estimates, the limitations of past practices, and new methods to address reliability generalization. We highlight best practices…

As evidence from evaluation and experimental studies continue to influence decision and policymaking, applied researchers and practitioners require tools to derive valid and credible inferences. Over the past several decades, research in causal inference has progressed with the development and application of propensity scores. Since their…

The aims of this reliability generalization study were to provide the overall alpha values of the California critical thinking disposition inventory (CCTDI) total score and subscales scores and investigate the characteristics of the studies that may be associated with the variability in the reliability values of the CCTDI total score and subscales…

This study aimed to provide meta-analytic reliability information of the Columbia-Suicide Severity Rating Scale (C-SSRS). We implemented systematic search procedures to 35 eligible studies (N = 23,247; Mage = 26.74 years) that reported reliability estimates. The synthesized average values of Cronbach's alpha were 0.88 (95% CI [0.85, 0.92]) for the…

Behavior rating scales measuring executive function have grown popular among school psychologists and other professions. To examine the generalizability of executive function rating scales, 42 parent-adolescent dyads, recruited from a school-based sample, completed the Behavior Rating Inventory of Executive Function, Second Edition and the…

Objective: Brief Self-Control Scale (BSCS) is the most widely used instrument to assess self-control. The purpose of this reliability generalization meta-analysis was to examine the degree to which consistency reliability coefficients for scores on the BSCS generalize across age groups and languages. Method: We included studies using the BSCS and…

Over the past ten years, propensity score methods have made an important contribution to improving generalizations from studies that do not select samples randomly from a population of inference. However, these methods require assumptions and recent work has considered the role of bounding approaches that provide a range of treatment impact…

This study examined the degree that reliability coefficients for scores on the PTGI generalize across participant and study characteristics. Meta-analytic procedures resulted in observed and predicted mean alpha coefficients ranging from acceptable to excellent and appeared to be largely unrelated to the participant characteristics included in our…

Employing G-theory and rater interviews, the study investigated how a high-stakes writing assessment procedure (i.e., a single-task, single-rater, and holistic scoring procedure) impacted the variability and reliability of its scores within the Turkish higher education context. Thirty-two essays written on two different writing tasks (i.e.,…

The purpose of this study was to estimate the overall reliability values for the scores produced by Runco Ideational Behavior Scale (RIBS) and explore the variability of RIBS score reliability across studies. To achieve this, a reliability generalization meta-analysis was carried out using the 86 Cronbach's alpha estimates obtained from 77 studies…

In recent times, Educational Data Mining and Learning Analytics have been abundantly used to model decision-making to improve teaching/learning ecosystems. However, the adaptation of student models in different domains/courses needs a balance between the generalization and context specificity to reduce the redundancy in creating domain-specific…

Functional plans in early childhood intervention need to include contextualized, meaningful, and measurable goals and include timelines and criteria for generalization. In addition, they must address children's and families' needs and priorities. The Routines-Based Interview has had a positive impact on the functionality of goals identified in the…

The ACTFL Assessment of Performance Toward Proficiency in Languages (AAPPL, https://www.actfl.n.d.org/assessments/k-12-assessments/aappl) assesses proficiency in 11 languages for students in grades 3 to 12 and is often used to award the Seal of Biliteracy. While arguments for the valid interpretation and uses of the AAPPL have previously been…

Jung and Won's (2018) review of elementary school ER found a lack of understanding of instructional practices for ER with young children. Other researchers have called for further studies into what effective classroom orchestration and interaction look like within ER classrooms (Ioannou & Makridou, 2018; Xia & Zhong, 2019). This study was…

The argument-based approach to validity (Kane, 2013) focuses on two steps: (1) making claims about the proposed interpretation and use of test scores as a coherent, interpretive argument; and (2) evaluating those claims based on theoretical and empirical evidence related to test performances and scores. This paper discusses the role of…