The Cross-Classified Mixed Effects Model (CCMEM) has been demonstrated to be a flexible framework for evaluating reliability by measurement specialists. Reliability can be estimated based on the variance components of the test scores. Built upon their accomplishment, this study extends the CCMEM to be used for evaluating validity evidence.…

The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an…

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper-based assessment (PBA). In the 2015 cycle, computer-based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an…

The use of assessment results to inform school accountability relies on the assumption that the test design appropriately represents the content and cognitive emphasis reflected in the state's standards. Since the passage of the Every Student Succeeds Act and the certification of accountability assessments through federal peer review practices,…

A multistage adaptive testing (MST) design was implemented for the Programme for the International Assessment of Adult Competencies (PIAAC) starting in 2012 for about 40 countries and has been implemented for the 2018 cycle of the Programme for International Student Assessment (PISA) for more than 80 countries. Using examples from PISA and PIAAC,…

One of the substantive changes in the 2014 Standards for Educational and Psychological Testing was the elevation of fairness in testing as a foundational element of practice in addition to validity and reliability. Previous research indicates that testing practices often do not align with professional standards and guidelines. Therefore, to raise…

There are many threats to validity in high-stakes achievement testing. One major threat is construct-irrelevant variance (CIV). This article defines CIV in the context of the contemporary, unitary view of validity and presents logical arguments, hypotheses, and documentation for a variety of CIV sources that commonly threaten interpretations of…

This article presents findings from two projects designed to improve evaluations of technical quality of alternate assessments for students with the most significant cognitive disabilities. We argue that assessment technical documents should allow for the evaluation of the construct validity of the alternate assessments following the traditions of…

Asserts that publishers of achievement tests are, for the most part, not in a position to obtain on their own any decent evidence about the consequences of uses made of their tests. Reasons why this is so are discussed, and what publishers can be expected to do is outlined. (SLD)

The relationships between ratings on the Idaho Alternate Assessment (IAA) for 116 students with significant disabilities and corresponding ratings for the same students on two norm-referenced teacher rating scales were examined to gain evidence about the validity of resulting IAA scores. To contextualize these findings, another group of 54…

Two achievement testing programs in New York City that have had and are having some consequences are described, and how the school system tries to determine and deal with these consequences is explored. Cooperation and open dialog between test publishers and the school district are making it possible to deal with the real-life problems of test…

Discusses some problems, pitfalls, and paradoxes that challenge measurement theory and practice, especially for K-12 achievement testing. Considers a number of technical issues, especially some related to reliability. Also discusses a number of practical or political issues related to validation and accountability. (SLD)

The author argues that scoring, reporting, and deriving final grades can be considerably assisted by using a computer. He also contends that the savings in time and the computer database formed will allow instructors to determine test quality and reflect on the quality of instruction. (BW)

Literature related to the multiple true-false (MTF) item format is reviewed. Each answer cluster of a MTF item may have several true items and the correctness of each is judged independently. MTF tests appear efficient and reliable, although they are a bit harder than multiple choice items for examinees. (SLD)

This paper discusses how to maintain the integrity of national nomative information for achievement tests when the test that is administered has been customized to satisfy local needs and is not a test that has been nationally normed. Alternative procedures for item selection and calibration are examined. (Author/LMO)