Standard setting is arguably one of the most subjective techniques in test development and psychometrics. The decisions when scores are compared to standards, however, are arguably the most consequential outcomes of testing. Providing licensure to practice in a profession has high stake consequences for the public. Denying graduation or forcing…

As a result of the upcoming Federal reauthorization of the Individuals With Disabilities Education Improvement Act (IDEA), practitioners and researchers have begun vigorously debating what constitutes evidence-based assessment for the identification of specific learning disability (SLD). This debate has resulted in strong support for a method that…

Developing and writing assessment items that measure teachers' knowledge is an intricate and complex undertaking. In this paper, we begin with an overview of what is known about measuring teacher knowledge. We then highlight the challenges inherent in creating assessment items that focus specifically on measuring teachers' specialised knowledge…

Compared to unidimensional item response models (IRMs), cognitive diagnostic models (CDMs) based on latent classes represent examinees' knowledge and item requirements using discrete structures. This study systematically examines the viability of retrofitting CDMs to IRM-based data with a linear attribute structure. The study utilizes a procedure…

In this article, we introduce a person-fit statistic called the hierarchy consistency index (HCI) to help detect misfitting item response vectors for tests developed and analyzed based on a cognitive model. The HCI ranges from -1.0 to 1.0, with values close to -1.0 indicating that students respond unexpectedly or differently from the responses…

This paper reviews the psychometric properties, treatment utility, and conceptual basis of instruments used to identify the functions of aberrant behaviors in people with developmental disabilities. Instruments include the Motivational Assessment Scale, Motivation Analysis Rating Scale, Functional Analysis Interview Form, and Functional Analysis…

The accuracy of 3 developmental screening tests administered to 89 young children was compared. The Battelle Developmental Inventory Screening Test was more accurate than the Academic Scale of the Developmental Profile-II and the Denver-II, identifying correctly 72% of children with difficulties and 76% of children without diagnoses. (Author/JDD)

The theoretical differences between the traditional definition of dimensionality and the more recently defined notion of essential dimensionality are presented. Monte Carlo simulations are used to demonstrate the utility of W. F. Stout's procedure to assess the essential unidimensionality of the latent space underlying a set of terms. The…

Confusion in the terminology used in criterion-referenced measurement specifications and development and standard setting and the attendant role of cut-off scores are shown to need practical clarification through psychometric research on test applications and consequences. (CM)

A method is proposed for sequential evaluation of reliability of psychometric instruments. Sample size is unfixed; a test statistic is computed after each person is sampled and a decision is made in each stage of the sampling process. Results from a series of Monte-Carlo experiments establish the method's efficiency. (SLD)

Three different applications of the testlet concept are presented, and the psychometric models most suitable for each application are described. Difficulties that testlets can help overcome include (1) context effects; (2) item ordering; and (3) content balancing. Implications for test construction are discussed. (SLD)

The idea that validity should be considered a property of inferences, rather than of assessments, has developed slowly over the past century. In early writings about the validity of educational assessments, validity was defined as a property of an assessment. The most common definition was that an assessment was valid to the extent that it…

Computerized adaptive testing (CAT), while well-grounded in psychometric theory, has had few large-scale applications for high-stakes, secure tests in the past. This is now changing as the cost of computing has declined rapidly. As is always true where theory is translated into practice, many practical issues arise. This paper discusses a number…

Political and legislative pressures have posed a number of measurement issues and challenges to the development of sound, valid voluntary national tests (VNTs). This paper focuses on what appear to be the most difficult technical issues related to the VNT proposed by President Clinton in 1997. Technical issues refer to psychometric issues, as…

Examines some problems of the Texas Assessment of Academic Skills (TAAS): multiple cutoff scores for passing the test and receiving a high school diploma, and artificially "tricky" items that disproportionately confuse language-minority students. Uses factor analysis and simulations to show ways to improve item selection for the TAAS.…