Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater-mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of…

This case study describes the process I engaged in to plan for and implement a design-based research project in an effort to explore what critical language study looked like in practice as well as how teachers and students responded to this instructional approach. In this case study, I focus, specifically, on the choice I made to take on the role…

This article presents a content analysis of the 2013 School Leadership Program (SLP) grants. SLP projects provide a unique opportunity for participants in the field to explore innovative leadership preparation and development and their impact on program participants, schools, school districts, and students. The article begins with an overview of…

This research gathered data from a sample of university students with disabilities (and a control group without disabilities) from two university campuses (one urban and the other suburban). The project examined the current work and school experiences, plans upon graduation, and understanding of major legislation designed to protect against…

The aim of the study was to investigate how raters come to their decisions when judging spoken vocabulary. Segmental rating was introduced to quantify raters' decision-making process. It is hoped that this simulated study brings fresh insight to future methodological considerations with spoken data. Twenty trainee raters assessed five Chinese…

The goal of this chapter is to recommend quality criteria to guide evaluators' selections of sampling designs when mixing approaches. First, we contextualize our discussion of quality criteria and sampling designs by discussing the concept of interpretive consistency and how it impacts sampling decisions. Embedded in this discussion are…

The continuously growing demand for program results has produced an increased need for evaluation capacity building (ECB). The "Integrative ECB Model" was developed to integrate concepts from existing ECB theory literature and to structure a synthesis of the empirical ECB literature. The study used a broad-based research synthesis method with…

This study addresses the central question "How do evaluators' background characteristics relate to their evaluation design choices?" Evaluators were provided with a fictitious description of a school-based program and asked to design an evaluation of that program. Relevant background characteristics such as level of experience,…

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However,…

Background: A music education program with a goal of enhancing cognitive development of preschool-aged children enrolled in local preschools is evaluated by The Evaluation Center at Western Michigan University. The budget for the evaluation was small, and therefore presented several challenges to the evaluation team. Purpose: Through a case study…

A simulation study was conducted in an attempt to examine how evaluators modify their evaluation design in response to differing stakeholder groups. In this study, evaluators were provided with a fictitious description of a school-based program. They were then asked to design an evaluation of the program. After the evaluation design decisions were…

The purpose of this study was to highlight the role of twin designs in understanding children's conversational interactions. Specifically, we (a) attempted to replicate the findings of genetic effects on children's conversational language use reported in DeThorne et al. (2008), and (b) examined whether the language used by examiners in their…

Rigorous research provides information that will allow ECE (Early Childhood Educator) programs to select interventions that have a scientifically based track record of effectiveness in increasing teachers' skills and teaching quality. This paper shares implementation and impact results as well as the lessons learned in conducting an evaluation of…

Previous efforts to investigate the equivalence of rating sources for job analysis ratings have reported conflicting results. In the present research, correlational and generalizability analyses were conducted to examine the equivalency of rating sources for over 70 state civil service job classifications. Incumbent and supervisor ratings (N=697)…

Consulting editors (N=101) of the Journal of Counseling Psychology and the Journal of Consulting and Clinical Psychology evaluated three versions, differing only with regard to level of statistical significance, of a research manuscript. Statistically nonsignificant and approach significance versions were more likely to be recommended for…