In this study, we examine "what went wrong" in our professional development program for encouraging cognitively demanding instruction, focusing on the difficulties we encountered in using an observational tool for evaluating this type of instruction and reaching inter-rater reliability. We do so through the lens of a discursive theory of…

Objective: The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these…

Emotional and behavioral problems (EBPs) frequently occur in young autistic children. Discrepancies between parents and other informants are common but can lead to uncertainty in formulation, diagnosis and care planning. This study aimed to explore child and informant characteristics are associated with reported child EBPs across settings.…

The Play Cycle Observation Method (PCOM) is an observational tool developed to focus on the process of play and has shown good reliability when watching videos of children playing. This study piloted use of the PCOM in 'real time' in a pre-school setting where 3-year-old children play. The results from two independent observers not familiar with…

Most teacher education assessments are criticized for lacking validity and reliability. This study describes the process of developing the Observation of Field Performance rubric to assess initial teacher candidates' classroom performance and establishing the content validity as well as reliability of the rubric. A panel of content area experts…

Following the pandemic-related school shutdowns in spring 2020, direct observations continued to be a necessary component of special education evaluations even when students were not present at school. As students began learning at home instead of in classrooms, the continued need for observational data likely compelled most educators to use video…

Teacher performance scores inform education leaders' management of teacher human resources. However, prior research has implied that different interpretations of performance criteria between teachers and their evaluators suppress teacher development. Although research has examined teacher perceptions of performance scores and compared teacher…

The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a…

Classroom management (CM) practices have a well-established, intuitive, and empirical connection with student academic, social, emotional, and behavioral outcomes. CM, defined as educator practices used to create supportive classroom environments, may be the implementation factor that is most impactful of the universal Tier I supports. Recognizing…

Classroom observation is a common approach to teacher evaluation. Yet, concerns about differences in rater judgment are widespread. Despite this concern, few researchers have examined the practical impact of such differences in rater judgments on teachers' judged effectiveness. This study fills that gap. Using data from a large-scale teacher…

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the…

Rater-mediated assessments are a common methodology for measuring persons, investigating rater behavior, and/or defining latent constructs. The purpose of this article is to provide a pedagogical framework for examining rater variability in the context of rater-mediated assessments using three distinct models. The first model is the observation…

The observation systems being used in all 50 states require administrators to learn to accurately and reliably score their teachers' instruction using standardized observation systems. Although the literature on observation systems is growing, relatively few studies have examined the outcomes of trainings focused on developing administrators'…

This study focuses on the use of systematic social observations (SSO) to measure crime prevention through environmental design (CPTED) and disorder. To improve knowledge about measurement issues in small area research, SSO is conducted by means of three different methods: in-situ, photographs, and Google Street View (GSV) imagery. By evaluating…

Traditionally, treatment adherence has been measured in outcome studies by highly trained, independent raters who code audio- or video-taped treatment sessions to understand therapists' adherence to preestablished treatment session components. Unfortunately, this method of assessing treatment adherence is time-, labor-, and cost-intensive, and…