Across the South, many states and school districts provide basic training for evaluators. But this basic training does not assure that evaluators will be prepared to provide teachers with classroom observations that are rigorous, reliable and personally relevant. High-quality evaluator training leads to improved instructional practice, both…

The purpose of this case study is twofold: it (1) explores multilingual raters' judgements regarding the accentedness and comprehensibility of second language speech and (2) examines how the raters' language backgrounds influence their judgements. In this study, six multilingual Singaporean raters judged the accentedness and comprehensibility of…

Culturally responsive evaluation contests that Western-derived evaluation methods represent a universal truth and promotes approaches that reflect the local context. Taking this approach, we examine how the method of contribution analysis may be modified to reflect a different worldview. We reframe contribution analysis using a Confucian lens as…

In this research report, we describe the design and empirical findings for a large-scale study of essay writing ability with approximately 2,500 high school students in Germany and Switzerland on the basis of 2 tasks with 2 associated prompts, each from a standardized writing assessment whose scoring involved both human and automated components.…

Funders and policymakers are increasingly recognizing the afterschool field for its vital role in supporting the social and emotional growth and academic achievement of school-age youth. Although this recognition is welcome, it often comes with increased expectations for high-quality research demonstrating the value of programming. To satisfy…

Threats to construct validity should be reduced to a minimum. If true, sources of bias, namely raters, items, tests as well as gender, age, race, language background, culture, and socio-economic status need to be spotted and removed. This study investigates raters' experience, language background, and the choice of essay prompt as potential…

Background: In our research project, we have developed a scoring rubric for a second language (L2) summary writing for English as a foreign language (EFL) students in Japanese universities. This study aimed to examine the applicability of our five-dimensional rubric, which features both analytic and holistic assessments, to classrooms in the EFL…

Despite the benefits of performance-based oral communication tests, a plethora of variables, as illustrated in Ockey and Li's (2015) model of oral communication assessment, can create construct-irrelevant variance in test scores. In relation to human participants in the oral communication tests, previous studies mostly focused on the direct effect…

Hiring an external evaluator is not right for every museum or every project. Evaluations are highly situational, grounded in specific times and places; each one is unique. The museum and the evaluator share equal responsibility in an evaluation's success, so it is worth investing time and effort to ensure that both are clear about the goals,…

This article discusses regression effects that are commonly observed in Angoff ratings where panelists tend to think that hard items are easier than they are and easy items are more difficult than they are in comparison to estimated item difficulties. Analyses of data from two credentialing exams illustrate these regression effects and the…

This mixed methods study investigates the juncture of dance science, somatics, and contemporary modern dance training, and how they intertwine with learning processes and skill execution through transfer theory. The effects of a novel somatics training program are analyzed through six case studies using data reduction and interpretation of…

The importance of engaging in high quality program evaluation is a generally accepted principle underscored by external pressure from funders. High quality evaluation necessarily begins with good evaluation planning. This paper outlines Evolutionary Evaluation and specifically the Systems Evaluation Protocol, an approach that emphasizes…

Contribution: This paper shows how significantly computer science instructors can disagree when grading code writing questions in introductory computer science (CS1) exams. Certain solutions are considered "very good" by some instructors and "very bad" by others. The study identifies four factors as possible contributors to…

Recent technology advancements have increased the prospects for automated spoken language technology to provide feedback on speaking performance. In this study we examined user perceptions of using an automated feedback system for preparing for the TOEFL iBT® test. Test takers and language teachers evaluated three types of machine-generated…

Over the past decade, researchers, policy makers, and practitioners have increasingly come to recognize the importance of quality in early care and education. Along with this recognition is compelling evidence that sound administrative practices help ensure high-quality learning opportunities for young children. Since its first publication in…