Proposed is a new method of standard setting referred to as response vector for mastery (RVM) method. Under the RVM method, the task of panelists that participate in the standard setting process does not involve conceptualization of a borderline examinee and probability judgments as it is the case with the Angoff and bookmark methods. Also, the…

This article discusses visual techniques for detecting test items that would be optimal to be selected to the final compilation on the one hand and, on the other hand, to out-select those items that would lower the quality of the compilation. Some classic visual tools are discussed, first, in a practical manner in diagnosing the logical,…

To evaluate the fit of a confirmatory factor analysis model, researchers often rely on fit indices such as SRMR, RMSEA, and CFI. These indices are frequently compared to benchmark values of 0.08, 0.06, and 0.96, respectively, established by Hu and Bentler (1999). However, these indices are affected by model characteristics and their sensitivity to…

This report describes a calibration, scoring, and reporting method applicable to an assessment requiring aggregation at multiple levels. We demonstrate the method using an example of an assessment that targets a small and diverse population, an English language proficiency assessment for English learners with the most significant cognitive…

In this digital ITEMS module, Dr. Michael Bunch provides an in-depth, step-by-step look at how standard setting is done. It does not focus on any specific procedure or methodology (e.g., modified Angoff, bookmark, and body of work) but on the practical tasks that must be completed for any standard setting activity. Dr. Bunch carries the…

Background/Context: Pass/fail standardized exams frequently selectively rescore failing exams and retest failing examinees. This practice distorts the test score distribution and can confuse those who do analysis on these distributions. In 2011, the Wall Street Journal showed large discontinuities in the New York City Regent test score…

Beginning with the 2020/21 school year, the Massachusetts Department of Elementary and Secondary Education (DESE) began an ongoing effort to collect and analyze literacy screening assessment data from schools and districts participating in certain state grants to inform improvement efforts. Grantee schools and districts that provide literacy…

Research suggests that kindergarten fine motor (FM) and visual motor (VM) skills predict later school performance. Being able to identify if gaps exist in FM/VM readiness could inform FM/VM programming in the early years. The Early Development Instrument is used to assess school readiness in Canada and other countries. Through a Delphi method, a…

The redesigned "TOEIC Bridge"® tests are designed to measure the reading, listening, speaking, and writing proficiency of beginning to low-intermediate English learners in the context of everyday adult life. This report describes the comprehensive and multifaceted process used to enhance the meaningfulness of TOEIC Bridge test score…

Standard setting is the process of identifying the point(s) on a scale that serve to differentiate between individuals of distinct proficiency levels. While standard setting is ultimately a policy decision, most of the process is carried out by subject matter experts who are tasked with reconciling item-level or examinee-level information (e.g.…

College and university admissions officers are the gatekeepers who decide which individuals may join the "country club" of higher education. As such, the decisions they make have a significant impact on the lives of individuals, especially those who are willing to travel across the globe and risk living in an unfamiliar country in order…

Treatment effect estimates from a "regression discontinuity design" (RDD) have high internal validity. However, the arguments that support the design apply to a subpopulation that is narrower and usually different from the population of substantive interest in evaluation research. The disconnect between RDD population and the…

Assessment is fundamental to school psychology, but its purpose has shifted from making predictions about children to improving outcomes for children. This commentary on the special issue focuses on screening and progress-monitoring decisions that can be used to solve student problems. We outline several psychometric and practical issues that…

For many students, developmental education is the largest obstacle to college success, hindering progress before they ever enroll in a college-level course. Postsecondary practitioners and researchers have been re-examining and challenging many traditional elements of developmental education from placement strategies to the sequence of courses…

The International English Language Testing System (IELTS) standardized English language proficiency (ELP) exam is widely accepted proof of ELP at Canadian universities. The majority of Canadian universities set very similar IELTS cut scores for admission; however, these differ from the IELTS-recommended minimum cut scores. The use of the IELTS…