In this digital ITEMS module, Dr. Stella Kim provides an overview of multidimensional item response theory (MIRT) equating. Traditional unidimensional item response theory (IRT) equating methods impose the sometimes untenable restriction on data that only a single ability is assessed. This module discusses potential sources of multidimensionality…

With the increasing use of automated scores in operational testing settings comes the need to understand the ways in which they can yield biased and unfair results. In this paper, we provide a brief survey of some of the ways in which the predictive methods used in automated scoring can lead to biased, and thus unfair automated scores. After…

We describe the development of 236 children with Angelman syndrome (AS) using the Bayley Scales of Infant and Toddler Development, Third Edition. Multilevel linear mixed modeling approaches were used to explore differences between molecular subtypes and over time. Individuals with AS continue to make slow gains in development through at least age…

Our study examines the differential item functioning of the Counselor Competencies Scale--Revised (CCS-R) scores due to respondents' gender, the type of evaluation, and a combination of these two variables using a large sample (N = 1614). Implications of the findings are offered to inform counselor educators and supervisors using the CCS-R and…

In order to detect a wide range of aberrant behaviors, it can be useful to incorporate information beyond the dichotomous item scores. In this paper, we extend the l[subscript z] and l*[subscript z] person-fit statistics so that unusual behavior in item scores and unusual behavior in item distractors can be used as indicators of aberrance. Through…

Validity is a fundamental consideration of test development and test evaluation. The purpose of this study is to define and reify three key aspects of validity and validation, namely test-score interpretation, test-score use, and the claims supporting interpretation and use. This study employed a Delphi methodology to explore how experts in…

The use of the aggregate scoring method for scoring concordance tests requires the weighting of test items to be derived from the performance of a group of experts who take the test under the same conditions as the examinees. However, the average score of experts constituting the reference panel remains a critical issue in the use of these tests.…

The part of responses that is absent in the nonequivalent groups with anchor test (NEAT) design can be managed to a planned missing scenario. In the context of small sample sizes, we present a machine learning (ML)-based imputation technique called chaining random forests (CRF) to perform equating tasks within the NEAT design. Specifically, seven…

This paper proposes to evaluate learning efficiency by implementing the flipped classroom and automatic source code evaluation based on the Kirkpatrick evaluation model in students of CS1 programming course. The experimentation was conducted with 82 students from two CS1 courses; an experimental group (EG = 56) and a control group (CG = 26). Each…

Though a substantial amount of research exists on imputing missing scores in educational assessments, there is little research on cases where responses or scores to an item are missing for all test takers. In this paper, we tackled the problem of imputing missing scores for tests for which the responses to an item are missing for all test takers.…

The Annual Professional Performance Review (APPR) system in New York State (NYS) has been called into question by educators since its adoption nearly 10 years ago, yet it remains the mandated evaluation system in NYS schools today. Much of the concern has been over changes such as assigning teachers final evaluation scores, as well as for the…

We investigate the effects of private schools on reading and numeracy scores using rich population data. Conditional on lagged test scores and narrowly defined neighborhood indicators, Catholic and non-Christian faith private schools on average raise test scores by 0.18 standard deviation or more relative to the average public school, while…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

This study was conducted to further research into the impact of student-group item parameter drift (SIPD) --referred to as subpopulation item parameter drift in previous research-- on ability estimates and proficiency classification accuracy when occurring in the discrimination parameter of a 2-PL item response theory (IRT) model. Using Monte…

This study aims to find out: firstly, the qualifications for developing teaching materials to evaluate observation-based history learning and secondly the level of students' critical thinking skills. The results of this research contribute to improving students' critical thinking skills through the development of teaching materials. This research…