Oral narrative assessments are important for diagnosis of language disorders in school-age children so scoring needs to be reliable and consistent. This study explored the impact of training on the variability of story grammar scores in children's oral narrative assessments scored by multiple raters. Fifty-one speech pathologists and 19 final-year…

Although the ethnography of speaking is one of the approaches used to analyze discourse (Schiffrin, 1994; Cameron, 2012), its benefits and uses can be applied in the field of language assessment when designing a drug-dispensing, classroom-based test task. In this article, pharmacy specialists and students functioning as members of the…

This article explores issues that had to be confronted over a three-year period in terms of two New York State funded grants for which I was the evaluator. Major, critical challenges are presented, described, and discussed. Specifically, the paper addresses issues related to lack of planning at the grant design stage that resulted in challenges…

The Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen's weighted kappa, the overall IRR estimate was 0.17 (poor strength of…

Professional competence of pre-service English teachers needs to be done to find effective ways in improving their teaching performance, which requires the proper implementation of ICT during online learning. This study aims at investigating the level of professional competence of pre-service English teachers in vocational high school in applying…

Recent advances in machine learning have made automated scoring of learner speech widespread, and yet validation research that provides support for applying automated scoring technology to assessment is still in its infancy. Both the educational measurement and language assessment communities have called for greater transparency in describing…

"Measure, Use, Improve! Data Use in Out-of-School Time" shares the experience and wisdom from a broad cross-section of out-of-school time professionals, ranging from internal evaluators, to funders, to researchers, to policy advocates. Key themes of the volume include building support for learning and evaluation within out-of-school time…

Recent public events have unnerved the country, touched the consciousness of the populace, and reveal the resurgence of the White supremacist movement, causing many citizens to reflect on race relations. Given the profound impact of racism deeply embedded in the fabric of this country and its complex and destructive influence on the lives of…

This report investigates whether the time between scoring sessions has an influence on operational and nonoperational scoring accuracy. The study evaluates raters' scoring accuracy on constructed-response essay responses for the "GRE"® General Test. Binomial linear mixed-effect models are presented that evaluate how the effect of various…

We propose a new framework for conceptualizing measures of instructed second language (L2) pronunciation performance according to three sets of parameters: (a) the constructs (focused on global vs. specific aspects of pronunciation), (b) the scoring method (human raters vs. acoustic analyses), and (c) the type of knowledge elicited (controlled vs.…

For agencies that deliver high-stakes Second Language (L2) proficiency exams, a research agenda has been undertaken for years to examine the role of rater, task, and rubric as sources of variability into their performance assessments (Lee, 2006; Sawaki & Sinharay, 2013; Xi, 2007; Xi & Mollaun, 2006). However, these challenges are more…

Background: Deficits in the ability to use language in social contexts, including storytelling skills, are observed across the autism spectrum. Development in machine-learning approaches may contribute to clinical psychology and psychiatry, given its potential to support decisions concerning the diagnosis and treatment of psychiatric conditions…

In language performance tests, raters are important as their scoring decisions determine which aspects of performance the scores represent; however, raters are considered as one of the potential sources contributing to unwanted variability in scores (Davis, 2012). Although a great number of studies have been conducted to unpack how rater…

In classroom learning, students need mathematical cognitive flexibility to be able to solve mathematical problems with the various ideas they express. To solve the problems, they must be able to grasp the problem, see it from various points of view, and should not be rigid thinking with one solving method. In fact, the students still lack the…

Purpose: This paper aims to determine how quality assurance practitioners in Vietnam describe their levels of professional knowledge and related skills, as well as identifying factors that interfere with the development of these skills. Design/methodology/approach: Semi-structured were conducted with 22 quality assurance practitioners who were…