We study how to automatically generate cloze questions from given texts to assess reading comprehension, where a cloze question consists of a stem with a blank space holder for the answer key, and three distractors for generating confusions. We present a generative method called CQG (Cloze Question Generator) for constructing cloze questions from…

The Canadian Agility and Movement Skill Assessment (CAMSA) is a valid and reliable circuit-based test of motor competence which can be used to assess children's skills in a live or recorded performance and then coded. We aimed to analyze the intrarater reliability of the CAMSA scores (total, time, and skill score) and time measured, by comparing…

In this study, I compare the accuracy of automatic speech recognition (ASR) transcription against two measures of intelligibility provided by human listeners. The data came from readings of five texts recorded by 15 language learners of French. Human understanding was gauged by (i) asking a group of 36 naïve first language (L1) speakers of French…

In 2004, Robert Stake published a provocative article that asked: "How Far Dare an Evaluator Go Toward Saving the World?" This question raises the issue of what role evaluators' values play in the conduct of evaluations. Following review of Stake's premises, I present value statements from a diverse group of 40 evaluators working with…

Since the No Child Left Behind Act (No Child Left Behind [NCLB], 2001) was enacted, the Bookmark method has been used in many state standard setting studies (Karantonis and Sireci; Zieky, Perie, and Livingston). The purpose of the current study is to evaluate the criticism that when panelists are presented with data during the Bookmark standard…

State and local education agencies across the country are prioritizing the goal of diversifying the teacher workforce. To further understand the challenges of diversifying the teacher pipeline, I investigate race and gender dynamics between teachers and school-based administrators, who are key decision-makers in hiring, evaluating, and retaining…

Massive open on-line courses (MOOCs) are effective and flexible resources to educate, train, and empower populations. Peer assessment (PA) provides a powerful pedagogical strategy to support educational activities and foster learners' success, also where a huge number of learners is involved. Item response theory (IRT) can model students'…

The award of a PhD degree is based on a rigorous examination process which is unique in that examiners can recommend that the degree is awarded subject to minor or major amendments to the thesis. In their examiner reports, examiners also include formative feedback that is intended to help the doctoral researchers improve their work. Using data…

Rater training is necessary to accurately evaluate fundamental motor skills among children with developmental disabilities (DD). The purpose of this pilot study was to examine the impact of an online rater training program for novice raters on scoring accuracy of the run and two-hand strike skills on the Test of Gross Motor Development-Third…

The current paper intends to exploit the Many Facet Rasch Model to investigate and compare the impact of situations (items) and raters on test takers' performance on the Written Discourse Completion Test (WDCT) and Discourse Self-Assessment Tests (DSAT). In this study, the participants were 110 English as a Foreign Language (EFL) students at…

As the development and application of large language models (LLMs) in physics education progress, the well-known AI-based chatbot ChatGPT4 has presented numerous opportunities for educational assessment. Investigating the potential of AI tools in practical educational assessment carries profound significance. This study explored the comparative…

The ethical guidelines for the American Evaluation Association and the principles of community-based participatory evaluation both state the importance of equitable stakeholder involvement. Regardless of the evaluation approach, however, evaluators are often confronted with gatekeepers, or those who control the access to stakeholders, information,…

Who should evaluate the originality and task-appropriateness of a given idea has been a perennial debate among psychologists of creativity. Here, we argue that the most relevant evaluator of a given idea depends crucially on the level of expertise of the person who generated it. To build this argument, we draw on two complimentary theoretical…

When is it appropriate, and consistent with ethical standards, for evaluators to modify reports in response to "requests" from clients to do so? Some evaluators may view any pressure to make any changes in their reports as an attack on their independence. While, clearly, it is not acceptable for evaluators to be asked to distort or…

Journals that adopt open peer review (OPR), where review reports of published articles are publicly available, provide an opportunity to study both review content characteristics and quantitative aspects of the overall review process. This study investigates two areas relevant to the quality assessment of manuscript reviews. First, do journal…