Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment.…

Recently, institutions have increased their online course offerings as well as their online degrees. With this significant growth in online offerings, assessment integrity becomes a concern. In response to this concern, many institutions have adopted the use of online proctoring services. The aim of using these online proctoring services is to…

The American Council on the Teaching of Foreign Languages (ACTFL) oral proficiency interview -- computer (OPIc) testing system represents an ambitious effort in language assessment: Assessing oral proficiency in over a dozen languages, on the same scale, from virtually anywhere at any time. Especially for users in contexts where multiple foreign…

The objective of this study was to develop Physics critical thinking skill test using computerized adaptive test (CAT) based on item response theory (IRT). This research was a development research using 4-D (define, design, develop, and disseminate). The content validity of the items was proven using Aiken's V. The test trial involved 252 students…

The intelligence community uses "structured analytic techniques" to help analysts think critically and avoid cognitive bias. However, little evidence exists of how techniques are applied and whether they are effective. We examined the use of the analysis of competing hypotheses (ACH)--a technique designed to reduce "confirmation…

Individuals trained in the use of cognitive tests should be able to complete an assessment without making administrative, scoring, or recording errors. However, an examination of 295 Wechsler protocols completed by graduate students and practicing school psychologists revealed that errors are the norm, not the exception. The most common errors…

Test score users often demand the reporting of subscores due to their potential diagnostic, remedial, and instructional benefits. Therefore, there is substantial pressure on testing programs to report subscores. However, professional standards require that subscores have to satisfy minimum quality standards before they can be reported. In this…

The natural experiment proposed in this article extracts three stories from boxes of "100 paper clips". The activity requires students to apply three lessons from inferential statistics, starting with a hypothesis test and including confidence intervals as well as tolerance intervals.

Given the importance of developing civically engaged and knowledgeable citizens, many have sought to improve the quality of students' civic education experiences. This article examines one particular effort spearheaded by the Joe Foss Institute, the Civic Education Initiative (CEI). Since the CEI was announced, 18 states have passed legislation…

This study developed a model of team mental models' influence on team adaptive performance at different stages. It proposed that mental models have different relationships with adaptive performance as a function of the performance stage when teams experience changes: early on (i.e., during situation assessment) more divergent mental models are…

The testing effect occurs when students, given information to learn and then practice during a test, perform better on a subsequent content post-test than students who restudy the information as a substitute for the practice test. The effect is often weaker or reversed if immediate rather than delayed post-tests are used. The weakening may be due…

Developer-friendly professional authoring tools have greatly simplified entertainment videogame development. However, this simplification had a limited impact on serious games, which require the active collaboration of developers with educators and other stakeholders. To address this issue, we present uAdventure, an easy-to-use game development…

Scoring of teachers' noticing responses is typically burdened with rater bias and reliance upon interrater consensus. The authors sought to make the scoring process more objective, equitable, and generalizable. The development process began with a description of response characteristics for each professional noticing component disconnected from…

Benefiting from item preknowledge (e.g., McLeod, Lewis, & Thissen, 2003) is a major type of fraudulent behavior during educational assessments. This paper suggests a new statistic that can be used for detecting the examinees who may have benefitted from item preknowledge using their response times. The statistic quantifies the difference in…

Intelligence Tutoring Systems (ITSs) focus on promoting knowledge acquisition, while providing relevant feedback during students' practice. Self-explanation practice is an effective method used to help students understand complex texts by leveraging comprehension. Our aim is to introduce a deep learning neural model for automatically scoring…