In this study, various statistical indexes of agreement were calculated using empirical data from a group of evaluators (n = 45) of early childhood teachers. The group of evaluators rated ten fictitious teacher profiles using the North Carolina Teacher Evaluation Process (NCTEP) rubric. The exact and adjacent agreement percentages were calculated…

The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient of Rater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance…

Parents offer unique information and are recognized as the main informants about children's social emotional functioning across time and situations. Inter-parental agreement was analysed with 100 preschoolers (M[subscript age] = 54.94 months) being rated independently by their mothers and fathers, using the Portuguese version of the Preschool and…

Early literacy screening tools are often a critical component of schools' intervention frameworks. As the use of screening tools becomes more prevalent for identifying students who may be at risk for learning difficulties and in need of additional instruction, it is important to continuously evaluate teachers' implementation of these screening…

Behavior rating scales are frequently used to assess social-emotional behaviors of children. While broadband behavior rating scales often measure similarly named constructs, it is unclear how consistently different instruments measure those constructs. Head Start teachers completed the preschool versions of the Behavior Assessment System for…

The Play Cycle Observation Method (PCOM) is an observational tool developed to focus on the process of play and has shown good reliability when watching videos of children playing. This study piloted use of the PCOM in 'real time' in a pre-school setting where 3-year-old children play. The results from two independent observers not familiar with…

Teacher evaluations are routinely conducted across the United States for licensure and professional development supports. However, there is limited research on the interrater reliability of these evaluation assessment systems, despite federal recommendations (Graham et al., 2012). This research explores the systematic approach to interrater…

Research Findings: This study explores the inter-rater reliability of WaKIDS, Washington State's kindergarten entry assessment (KEA). Specifically, we analyze (1) the extent to which teachers' assessments are in agreement with a master code, (2) how often inaccurate assessment decisions lead to misidentification of school readiness, and (3)…

Open-ended verbal creativity assessments are commonly administered in psychological research and in educational practice to elementary-aged children. Children's responses are then typically rated by teams of judges who are trained to identify original ideas, hopefully with a degree of inter-rater agreement. Even in cases where the judges are…

The objective of this paper is to present the conception and validation of the Quality of Educators' Observation and Planning Practices Scale (QEOPPS). The instrument assesses the quality of early childhood educators' observation practices (observing children, collecting information, using the collected information, paying attention to each child)…

Running Records provide a standardized method for recording and assessing students' oral reading behaviors and are excellent formative assessment tools to guide instructional decision-making. This study expands on prior Running Record reliability work by evaluating the extent to which external raters and teachers consistently assessed students'…

This study details the construction of parameters for generating subskill mastery math measures to be used for screening, intervention planning, progress monitoring, and proximal program evaluation. Parameters for generating assessment measures were built and tested to verify initial equivalence of generated measures using potential digits correct…

This study utilized a quantitative research approach with a survey design to determine whether a sample of K-8 principals in the state of Illinois, that rated teachers in a set of classroom instructional videos, exhibited inter-rater reliability since they all received the Illinois state-approved training and certification for teacher evaluators.…

This article describes the development of the Early Elementary Writing Rubric (EEWR), an analytic assessment designed to measure kindergarten and first-grade writing and inform educators' instruction. Crocker and Algina's (1986) approach to instrument development and validation was used as a guide to create and refine the writing measure. Study 1…

The purpose of this article is to provide evidence of the technical adequacy of the Assessment, Evaluation, and Programming System--Third Edition (AEPS-3). The AEPS has long been identified as one of the most psychometrically sound early childhood curriculum-based assessments. In this article, results of three studies of technical adequacy are…