Rater training is essential in ensuring the quality of constructed response scoring. Most of the current knowledge about rater training comes from experimental contexts with an emphasis on short-term effects. Few sources are available for empirical evidence on whether and how raters become more accurate as they gain scoring experiences or what…

This study examines the early-stage progress assessment for doctoral candidates at a British university through the perceptions of academic staff who acted as assessors. It was conducted in response to staff concerns about changes to the process that shortened the timescale for candidates to prepare and present their research for peer review. As…

Creativity is highly valued in both education and the workforce, but assessing and developing creativity can be difficult without psychometrically robust and affordable tools. The open-ended nature of creativity assessments has made them difficult to score, expensive, often imprecise, and therefore impractical for school- or district-wide use. To…

Virtual reality simulation has had a significant impact on training of psychomotor surgical skills, yet there is still a lack of work on its use to teach surgical decision making. This is particularly noteworthy given the recognized importance of decision making in achieving positive surgical outcomes. With the objective of filling this gap, we…

This study explored two assessment approaches to oral performances: analytical complexity, accuracy, and fluency (CAF) indices and human raters' evaluations. CAF indices are frequently used in second-language speaking (L2) research; however, because tasks are communicative and goal-oriented, the degree to which students achieve such communicative…

Scoring rubrics are known to be effective for assessing writing for both testing and classroom teaching purposes. How raters interpret the descriptors in a rubric can significantly impact the subsequent final score, and further, the descriptors may also color a rater's judgment of a student's writing quality. Little is known, however, about how…

The rise in the use of scientific evidence as a basis for educational policymaking has been a noticeable feature globally. In this study, we describe and discuss how educational research is used to make policy and governing evidence-based. To illustrate this, we use the Swedish Schools Inspectorate (SSI) as a case and focus on two of the processes…

The aim of the present study was to examine how social evaluation influences creative idea generation, and whether this effect develops with age. To do so, early adolescents, middle adolescents, and late adolescents performed a creative task either alone or under the supervision of an adult examiner. Three major findings emerged: 1) the social…

The self-report and think-aloud approaches are the two dominant methodologies to measure metacognition. This is problematic, since they generate respondent and confirmation biases, respectively. The Meta-Performance Test is an innovative battery, which evaluates metacognition based on the respondent's performance, mitigating the aforementioned…

This study investigates the perceived benefits to academics who assess HEA Fellowship applications in one institution. Results show that the assessors experience similar benefits from engaging with the continuing professional development (CPD) scheme as those reported by applicants. The data also indicate that the scheme nurtures a…

Communication in the real world often entails the interpretation, evaluation, and integration of content from different sources. However, it appears that the ability to integrate content into discourse has not been explicitly scored for in existing studies. This study operationalizes content integration in the analytic scoring of a…

Automated assessment using Natural Language Processing (NLP) has the potential to make English speaking assessments more reliable, authentic, and accessible. Yet without careful examination, NLP may exacerbate social prejudices based on gender or native language (L1). Current NLP-based assessments are prone to such biases, yet research and…

Science, Technology, Engineering, and Mathematics (STEM) education has become increasingly important in the US due to its influence on the nation's educational needs, the creation of a skilled labor force, and opportunities for more tech-savvy workers. However, the evaluation approaches and methodologies used in STEM education programs have come…

This paper provides an experimental study of interlanguage phonological characteristics of Chinese students learning Thai as a foreign language and the accentedness perceived by native Thai speakers. Both production and perception experiments were designed to see how Chinese students acoustically produced Thai final nasal consonants and how Thai…

Despite the increase in federal tiered evidence initiatives that require the use of rigorous evaluation designs, such as randomized experiments, there has been limited guidance in the evaluation literature on practical strategies to implement such studies successfully. This paper provides lessons learned in executing experiments in applied…