Taxonomies of cognitive demand are frequently used to ensure that assessment tasks include questions ranging from low to high cognitive demand. This paper investigates inter-rater agreement among four evaluators on the cognitive demand of the South African National Senior Certificate Life Sciences examinations after training, practice and…

Rater variability has always been identified as an important source of measurement error in performance assessment, especially for oral proficiency tests. Rater training is commonly used as a means for compensating various sources of rater variability and adjusting their assessment quality. However, there is little research regarding the nature of…

One common modification to the Angoff standard-setting method is to have panelists round their ratings to the nearest 0.05 or 0.10 instead of 0.01. Several reasons have been offered as to why it may make sense to have panelists round their ratings to the nearest 0.05 or 0.10. In this article, we examine one reason that has been suggested, which is…

Classroom observations remain the predominant data source used in teacher evaluations, but little is known about how rater characteristics may affect teachers' scores. For special educators, whose instructional practice requires specialized knowledge and skills, school administrators (i.e., the raters) without experience in special education…

Implementation of research-based, Tier 1 behavior management strategies can be monitored to provide data-driven feedback and in support of integrity. The "Measure of Active Supervision and Interaction" (MASI) was developed to measure four behavior management practices (i.e., Praise, Correction, References to Behavior Expectations, Active…

Scores from noncognitive measures are increasingly valued for their utility in helping to inform postsecondary admissions decisions. However, their use has presented challenges because of faking, response biases, or subjectivity, which standardized third-party evaluations (TPEs) can help minimize. Analysts and researchers using TPEs, however, need…

Students with Emotional and Behavioural Disorders (EBD) have the poorest academic and social outcomes across the general and special education student populations, and are among the most likely to receive instruction in self-contained special education classrooms hallmarked by small teacher-student ratios, frequent transitions, extreme student…

In face-to-face surveys, interviewer observations are a cost-effective source of paradata for nonresponse adjustment of survey estimates and responsive survey designs. Unfortunately, recent studies have suggested that the accuracy of these observations can vary substantially among interviewers, even after controlling for household-, area-, and…

Bilingual literature for children is valuable in encouraging literacy in second language learners. Stories can enhance vocabulary and language abilities, learning encounters, subject content, social aptitude, and other skills in the early reader through text as well as illustrations. This paper explores issues in selecting quality dual language…

This study aimed to assess the validity and reliability of jump assessments using the "MyJump2" application. Eleven junior athletes (15 ± 1.4 years) performed five countermovement (CMJ) and drop jumps (DJ) measured simultaneously by a force platform and "MyJump2." Additionally, intra- and inter-day reliability was assessed over…

Purpose: This study presents a novel methodology/process for assessing inclusion of theoretically-based implementation factors within available adoption-ready health promotion programs. Methods: Classroom-based physical activity (CBPA) programs were used as an example to describe the process. Our team selected an implementation science framework…

Children from low-income households are at risk for entering school behind their more economically advantaged peers across major domains of school readiness. The Head Start program represents the federal government's response to these achievement gaps by mandating the use of scientifically based assessments and curricula to provide children with…

Though treatment integrity measurement is important for research intended to promote social and behavioral outcomes of children at risk for emotional and behavioral disorders (EBDs) in early childhood settings, measurement gaps exist in the field. This paper reports on the development and preliminary psychometric assessment of the treatment…

Drawing parallels to classical test theory, this article clarifies the difference between rater accuracy and reliability and demonstrates how category marginal frequencies affect rater agreement and Cohen's kappa. Category assignment paradigms are developed: comparing raters to a standard (index) versus comparing two raters to one another…

Assessing global interrater agreement is difficult as most published indices are affected by the presence of mixtures of agreements and disagreements. A previously proposed method was shown to be specifically sensitive to global agreement, excluding mixtures, but also negatively biased. Here, we propose two alternatives in an attempt to find what…