Objective: This study aimed to develop and test a scale for measuring the quality of blended learning models in higher education. Methods: This research adopts a sequential mixed-method approach to construct a new measurement scale. The first phase consisted of the inductive approach to identify the items, followed by exploratory factor analysis.…

Research in the social and behavioral sciences relies on a wide range of experimental and quasi-experimental designs to estimate the causal effects of specific programs, policies, and events. In this paper we highlight measurement issues relevant to evaluating the validity of causal estimation and generalization. These issues impact all four…

Feedback on student assignments plays a crucial role in steering students toward academic success. To provide feedback more promptly and efficiently, researchers are actively exploring the use of large language models (LLMs) to automatically generate feedback on student artifacts. Although the generated feedback is highly fluent, coherent, and…

The observation systems being used in all 50 states require administrators to learn to accurately and reliably score their teachers' instruction using standardized observation systems. Although the literature on observation systems is growing, relatively few studies have examined the outcomes of trainings focused on developing administrators'…

As diagnostic classification models become more widely used in large-scale operational assessments, we must give consideration to the methods for estimating and reporting reliability. Researchers must explore alternatives to traditional reliability methods that are consistent with the design, scoring, and reporting levels of diagnostic assessment…

There is a critical need for high-quality clinical outcome assessments to capture the important aspects of communication ability of individuals with Angelman syndrome (AS). To center the perspective of caregivers, our team developed the novel Observer-Reported Communication Ability (ORCA) measure using best practice guidelines, with the goal of…

Background: Appropriate diagnosis of sleep problems is crucial, given the importance of sleep in childhood development. The Sleep Self-Report Scale (SSRS) is used to assess children's sleep problems in the United States and Spain, and this study aimed to expand the usability of this instrument by evaluating its validity and reliability in Turkish…

In this study, we examine "what went wrong" in our professional development program for encouraging cognitively demanding instruction, focusing on the difficulties we encountered in using an observational tool for evaluating this type of instruction and reaching inter-rater reliability. We do so through the lens of a discursive theory of…

A principals' school management imaginative capability is the cornerstone of visionary leadership. Based on the imagination theory, this study constructed a Principal's School Management Imaginative Capability Scale (PSMICS). Questionnaires were conducted through stratified random sampling. Thirteen hundred and two valid samples were obtained.…

In the context of virtual learning communities (VLCs), where the participants may not know each other, it is necessary to have a mechanism to help when deciding who to work with and what reliable contents and information sources are. This study aims to design a generic trust model, named T-VLC, applicable to VLCs, which can be adapted to different…

Computer-Aided Detection (CAD) has been proposed to help operators search for cancers in mammograms. Previous studies have found that although accurate CAD leads to an improvement in cancer detection, inaccurate CAD leads to an increase in both missed cancers and false alarms. This is known as the over-reliance effect. We investigated whether…

Objective: The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these…

The aims of this reliability generalization study were to provide the overall alpha values of the California critical thinking disposition inventory (CCTDI) total score and subscales scores and investigate the characteristics of the studies that may be associated with the variability in the reliability values of the CCTDI total score and subscales…

The limitations of Cohen's ? are reviewed and an alternative G-index is recommended for assessing nominal-scale agreement. Maximum likelihood estimates, standard errors, and confidence intervals for a two-rater G-index are derived for one-group and two-group designs. A new G-index of agreement for multirater designs is proposed. Statistical…

The purpose of this study was to investigate a new way of evaluating interrater reliability that can allow one to determine if two raters differ with respect to their rating on a polytomous rating scale or constructed response item. Specifically, differential item functioning (DIF) analyses were used to assess interrater reliability and compared…