Both school and district administrators use the results of standardized, large-scale tests to inform decisions about the need for, or success of, educational programs and interventions. However, test results at the school level are subject to random fluctuations due to changes in cohort, test items, and other factors outside of the school's…

Generalizability theory is a modern, powerful, and broad framework used to assess the reliability, or dependability, of measurements. While there exist classic works that explain the basic concepts and mathematical foundations of the method, there is currently a lack of resources addressing computational resources for those researchers wishing to…

To meet recent accountability mandates, school districts are implementing assessment frameworks to document teachers' effectiveness. Observational assessments play a key role in this process, albeit without compelling evidence of their psychometric rigor. Using a sample of kindergarten teachers, we employed Generalizability theory to investigate…

The aim of this study is to develop a scoring rubric to assess primary school students' problem posing skills. The rubric including five dimensions namely solvability, reasonability, mathematical structure, context and language was used. The raters scored the students' problem posing skills both with and without the scoring rubric to test the…

Heightened accountability pressures and an increased emphasis on teaching quality have directed scholarly attention to scrutinizing instruction, particularly with respect to issues of validity and reliability. However, these attempts have largely been directed toward "core" content areas and investigated generic or content-specific…

Reliability and differential item functioning (DIF) analyses were conducted on testlets displaying local item dependence in this study. The data set employed in the research was obtained from the answers given by 1,500 students to the 20 items included in six testlets given in English Proficiency Exam by the School of Foreign Languages of a state…

This study used generalizability theory to measure reliability on the Recognizing Effective Special Education Teachers (RESET) observation tool designed to evaluate special education teacher effectiveness. At the time of this study, the RESET tool included three evidence-based instructional practices (direct, explicit instruction; whole-group…

We report findings of two exploratory validation studies of a revised instrument: the "Student Risk Screening Scale for Early Childhood" version (SRSS-EC). The SRSS-EC was modified to reflect characteristics of externalizing and internalizing behaviors manifested by preschool-age children. In Study 1, we explored the reliability of…

Assessing conceptual understanding of foundational topics before instruction on higher-order concepts can provide chemical educators with information to aid instructional design. This study provides an instrument that can be used to identify students' alternative conceptions regarding thermochemistry concepts. The Thermochemistry Concept Inventory…

Although federal regulations require testing students with severe cognitive disabilities, there is little guidance regarding how technical quality should be established. It is known that challenges exist with documentation of the reliability of scores for alternate assessments. Typical measures of reliability do little in modeling multiple sources…

This paper reports the results of interrater agreement analyses on a pilot special education teacher evaluation instrument, the Recognizing Effective Special Education Teachers (RESET) Observation Tool (OT). Using evidence-based instructional practices as the basis for the evaluation, the RESET OT is designed for the spectrum of different…

This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers' cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a "Breakdown" is a test…

Addresses generalizability theory (GT), which offers a flexible framework for assessing dependability of measurement. GT allows for consideration of multiple sources of error, allowing investigators to assess the overall impact of measurement error. Illustrative analyses demonstrate the special advantages of GT for planning studies in which…

A pilot study was conducted to evaluate and improve the rating procedure proposed for use in a research effort designed to assess the essay writing ability of college sophomores. Generalizability theory and the Many-Facet Rasch Model were each used to (a) estimate potential sources of error in the rating, (b) to obtain reliability estimates, and…

This paper argues that special procedures for constructing assessment tools containing performance assessment tasks are unnecessary and that current test methodology can easily be generalized to complex performance assessment tasks without destroying the desirable characteristics of those tasks. Reasonable statistical requirements for sound…