How can we test whether state-of-the-art generative models, such as Blender and GPT-3, are good AI teachers, capable of replying to a student in an educational dialogue? Designing an AI teacher test is challenging: although evaluation methods are much-needed, there is no off-the-shelf solution to measuring pedagogical ability. This paper reports…

Research on self-efficacy has provided evidence that it is a moderating factor that positively impacts students' choices to pursue and persist in engineering. Engineering graphics is seen as the preferred method of communication for the profession, yet to date no instrument is available that measures students' self-efficacy as it relates to…

Colleges and universities are being asked by numerous sources to provide assurance of learning assessments of their students and programs. Colleges of business have responded by using a plethora of assessment tools, including the Major Field Test in Business. In this article, the authors show that the use of the Major Field Test in Business for…

Educational and psychological constructs are normally measured by multifaceted dimensions. The measured construct is defined and measured by a set of related subdomains. A bifactor model can accurately describe such data with both the measured construct and the related subdomains. However, a limitation of the bifactor model is the orthogonality…

An automated item selection procedure in Mokken scale analysis partitions a set of items into one or more Mokken scales, if the data allow. Two algorithms are available that pursue the same goal of selecting Mokken scales of maximum length: Mokken's original automated item selection procedure (AISP) and a genetic algorithm (GA). Minimum…

Automated test assembly (ATA) has been an area of prolific psychometric research. Although ATA methodology is well developed for unidimensional models, its application alongside cognitive diagnosis models (CDMs) is a burgeoning topic. Two suggested procedures for combining ATA and CDMs are to maximize the cognitive diagnostic index and to use a…

Bayesian network models offer a large degree of flexibility for modeling dependence among observables (item outcome variables) from the same task, which may be dependent. This article explores four design patterns for modeling locally dependent observations: (a) no context--ignores dependence among observables; (b) compensatory context--introduces…

This simulation study compared the performance of two multilevel measurement testlet (MMMT) models: Beretvas and Walker's (2008) two-level MMMT model and Jiao, Wang, and Kamata's (2005) three-level model. Several conditions were manipulated (including testlet length, sample size, and the pattern of the testlet effects) to assess the impact on the…

The early stage of computerized adaptive testing (CAT) refers to the phase of the trait estimation during the administration of only a few items. This phase can be characterized by bias and instability of estimation. In this study, an item selection criterion is introduced in an attempt to lessen this instability: the D-optimality criterion. A…

Conceptualized eight different types of measurement models for a test composed of testlets and studied the goodness of fit of those models to data using data from the Iowa Tests of Basic Skills and simulated data. The essentially tau-equivalent model and the congeneric model provided worse model fit than the other measurement models. (SLD)

Although cognitive diagnostic models (CDMs) can be useful in the analysis and interpretation of existing tests, little has been developed to specify how one might construct a good test using aspects of the CDMs. This article discusses the derivation of a general CDM index based on Kullback-Leibler information that will serve as a measure of how…

Defines and demonstrates a framework for studying differential item functioning and differential test functioning for tests that are intended to be multidimensional. The procedure, which is illustrated with simulated data, is an extension of the unidimensional differential functioning of items and tests approach (N. Raju, W. van der Linden, and P.…

It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all new and currently considered computer-based tests. In addition to developing new models, researchers will need to give some attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized…

The purpose of this study was to test a specific hypothesis with regard to how cloze retrieval takes place once semantic constraints are recognized. Results from human subjects and computer simulation suggested that increase in latency between the two- and three-semantic-constraint conditions was not artifactual. (JAZ)