The calculation of composite indicators and the derivation of respective rankings is a common method used to benchmark countries or regions. However, although the statistical robustness of these rankings is often criticised, they often still spark off heated political debate. Here, we assess the sensitivity of the province ranking published by the…

The present study utilized both the IRT-LR (item response theory likelihood ratio) and a series of CFA (confirmatory factor analysis) multi-sample analyses to systematically examine the relationships between DIF (differential item functioning) and DTF (differential test functioning) with a random sample of 15 000 Korean examinees. Specifically,…

Neuropsychological tests have limited sensitivity in identifying subtle residual cognitive impairments in patients with good medical recovery from head injury and post-concussive syndrome. Detecting and characterizing residual "real life" cognitive difficulties can be problematic for treatment purposes. This study investigated the usefulness of a…

There are usually many testing activities going on in a school, with different tests serving different purposes, thus organization and planning are key in creating an efficient system in assessing the most important educational objectives. In the ideal case, an assessment system will be able to inform on student learning, instruction and…

The goal of this research project was to investigate the use of dynamic assessment to increase equity, fairness, and accuracy in the testing of abilities and achievement. Dynamic tests have been found to reveal developing expertise in underrepresented minorities around the world that is not revealed by conventional static tests. The…

The AP Course Audit utilizes a criterion-based professional judgment method of analysis within a nested multi-step review process. The overall goal of the methodology is to yield a final judgment on each syllabus that is ultimately valid. While reviewer consistency is an important consideration, the most important goal is to reach a final judgment…

This article addresses a pervasive need in the area of videoconference-supported distance language learning: task design. On the basis of Chapelle's (2001) criteria for CALL task appropriateness, this article proposes a set of criteria for evaluating videoconferencing-based tasks which examine such aspects of a task as practicality,…

The IRT (Item Response Theory) has been studied and applied in computer-based test for decades. However, almost of all these existing studies evaluated focus merely on test questions with text-based (or static text/graphic) type of presentation form illustrated exclusively. In this paper, we present our study on test questions using both…

This paper introduces criterion-referenced tests (CRTs), compares them with norm-referenced tests (NRTs), discusses how they can be evaluated and revised, and presents a study of an actual class and textbook test evaluation using CRTs. NRTs have dominated testing methodology since the mid-1970s; an example is the Test of English as a Foreign…

A study was designed to provide recommendations regarding the use of the achievement levels set in 1992 for reporting National Assessment of Educational Progress (NAEP) reading results in 1994 and in future NAEP reading assessments. Two procedures were used: the Item Difficulty Categorization (IDC) procedure involved an evaluation of the…

This paper presents a technique for applying the Rule Space Model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically rich domain. Responses of 122 architects to 22 architecture test items developed to assess a range of architectural knowledge were analyzed using Rule Space. Verbal protocol analysis guided the construction of a…

This technical report describes the development of the College-Level Academic Skills Test (CLAST), an instrument designed to measure Florida college students' achievement of the computation and communication skills expected by the completion of their sophomore year. Section I covers CLAST's background and purpose, the requirement that all students…

A study was undertaken to identify the prerequisite relations (or hierarchies among the items) existing in the item responses of a sample of 86 foreign students who took the Test of English as a Foreign Language (TOEFL) vocabulary and reading comprehension test, Form 3JTF1. The form contains 30 vocabulary items and 30 reading comprehension items.…

In item response theory (IRT), preequating depends upon item parameter estimate invariance. Three separate simulations, all using the unidimensional three-parameter logistic item response model, were conducted to study the impact of the following variables on preequating: (1) mean differences in ability; (2) multidimensionality in the data; and…

The purpose of this study was to determine whether item difficulty is significantly affected by language difficulty and response set convergence. Language difficulty was varied by increasing sentence (stem) length, increasing syntactic complexity, and substituting uncommon words for more familiar terms in the item stem. Item wording ranged from…