This article considers the challenges of using artificial intelligence (AI) and machine learning (ML) to assist high-stakes standardised assessment. It focuses on the detrimental effect that even state-of-the-art AI and ML systems could have on the validity of national exams of secondary education, and how lower validity would negatively affect…

An approximation to the likelihood for the generalized linear models with random coefficients is derived and is the basis for an approximate Fisher scoring algorithm. The method is illustrated on the logistic regression model for one-way classification, but it has an extension to the class of generalized linear models and to more complex data…

This study is a critical evaluation of the roles for coding and scoring of missing responses to multiple-choice items in educational tests. The focus is on tests in which the test-takers have little or no motivation; in such tests omitting and not reaching (as classified by the currently adopted operational rules) is quite frequent. Data from the…

In conventional practice, most educators and educational researchers score cognitive tests using a dichotomous right-wrong scoring system. Although simple and straightforward, this method does not take into consideration other factors, such as partial knowledge or guessing tendencies and abilities. This paper discusses alternative scoring models:…

A marginal maximum likelihood estimation algorithm is presented for the ordered partition model of M. Wilson that does not require the set of available responses to be fully ordered. The model and its estimation algorithm are illustrated in a comparison of alternative scoring schemes for open-ended science items. (SLD)

Research on automated scoring is extended by comparing alternative automated systems for scoring a computer simulation of physicians' patient management skills. A regression-based system is more highly correlated with experts' evaluations than a system that uses complex rules to map performances into score levels, but both approaches are feasible.…

Three different applications of the testlet concept are presented, and the psychometric models most suitable for each application are described. Difficulties that testlets can help overcome include (1) context effects; (2) item ordering; and (3) content balancing. Implications for test construction are discussed. (SLD)

This project explored the possible applications of new technology to standardized testing. Because the multimedia approach is particularly promising for music, the Graduate Record Examination (GRE) music test was chosen as the context for this project. The basic system for administering the test and recording the results was written using…

A two-level model for factor analysis is defined, and formulas for a scoring algorithm for this model are derived. A simple noniterative method based on decomposition of total sums of the squares and cross-products is discussed and illustrated with simulated data and data from the Second International Mathematics Study. (SLD)

Describes an automated scoring algorithm for a computer-based simulation examination of physicians' patient-management skills. Results with 280 medical students show that scores produced using this algorithm are highly correlated to actual clinician ratings. Scores were also effective in discriminating between case performance judged passing or…

A paradigm for the recalibration of teacher-made assessment that assesses and evaluates in one operation is formulated. The effort to make the classroom the primary source of educational research activity is contingent on redefining educational research as empirical and not experimental. This emphasizes that the empirical analysis of instructional…

Performance rating systems frequently use multiple raters in order to improve the reliability of ratings. However, unless all candidates are rated by the same raters, some candidates will be at an unfair advantage or disadvantage solely because they were rated by more stringent or lenient raters. To obtain fair and accurate evaluations of…

Methods of computerized adaptive testing using conventional scoring methods in order to develop a computerized placement test for the Test of English as a Foreign Language (TOEFL) were studied. As a consequence of simulation studies during the first phase of the study, the multilevel testing paradigm was adopted to produce three test levels…