This study investigated the effectiveness of statistical adjustments applied to rater bias in many-facet Rasch analysis. Some changes were first made in the dataset that did not include "rater × examinee" bias to cause to have "rater × examinee" bias. Later, bias adjustment was applied to rater bias included in the data file,…

Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational Progress (NAEP) data mining competition. The…

Teacher observations have become a national education phenomenon, mandated by federal policies and promoted by philanthropists. They are crucial components of teacher evaluation systems that often have high stakes for teachers and school systems, but have sparked little innovation. Recent calls to make teacher evaluations better, faster, and…

Researchers have expressed concern about "implementation" fidelity in intervention research but have not extended that concern to "assessment" fidelity, or the extent to which pre-/posttests are administered and interpreted as intended. When studying reading interventions, data gathering heavily influences the identification of…

The alignment among standards, assessments, and teachers' instruction is an essential element of standards-based educational reforms. The Surveys of Enacted Curriculum (SEC) is the only common tool that can be used to measure the alignment among all three of these sources (Martone & Sireci, 2009). Prior SEC alignment work has been limited by…

We investigated the effects of expressive and unexpressive conducting on secondary school band members' and experts' audio evaluations of band performance expressivity. A conductor, who demonstrated either expressive or unexpressive conducting techniques, led both an eighth-grade and a high school band in four separate "run-throughs" of…

We evaluated the reliability and validity of two oral reading fluency scores for 1-minute equated passages: median score and mean score. These scores were calculated from measures of reading fluency administered up to five times over the school year to students in grades six to eight (n = 1,317). Both scores were highly reliable with strong…

Evaluators use their eyes to see what is there, whether it is intended or not. But they use their test instruments to measure what is intended, whether it is there or not. Evaluators have been broadening their repertoire of instruments for years: curriculum-embedded tests, observer checklists, audiotape recorders, videotape recorders, unobtrusive…

School program evaluation researchers face a set of overlapping questions concerning our roles in the field: For the sake of "the data" and in quest of "the truth," am I a shrewd researcher before all else? For the sake of community-building and establishing respectful, reciprocal relationships with my school partners, am I first a gracious school…

All evaluators face the challenge of striving to adhere to the highest possible standards of ethical conduct. Translating the AEA's Guiding Principles and the Joint Committee's Program Evaluation Standards into everyday practice, however, can be a complex, uncertain, and frustrating endeavor. Moreover, acting in an ethical fashion can require…

Legislation passed during the 2001 Iowa legislative session established the Student Achievement and Teacher Quality Program, Iowa Code Section 284.12(1). This legislation requires the Iowa Department of Education (DE) to annually report the statewide progress on the following: student achievement scores in mathematics and reading at the fourth and…

Legislation passed during the 2001 Iowa legislative session established the Student Achievement and Teacher Quality Program, Senate File 476. This legislation requires the Iowa Department of Education (DE) to annually report the statewide progress on the following: (1) Student achievement scores in mathematics and reading at the fourth and eighth…

This is the first report on the evaluation of the Inquiry Based Science and Technology Education Program (IN-STEP), an innovative and ambitious science education initiative for lower secondary schools being undertaken by a public-private partnership in Thailand funded by MSD-Thailand, an affiliate of Merck & Co. IN-STEP is a public-private…