This report summarizes the work conducted for the Artificial Intelligence Measurement System (AIMS) Project which was undertaken as an exploration of methodology to consider how the effects of artificial intelligence systems could be compared to human performance. The research covered four areas of inquiry: (1) natural language processing and…

Among opportunities to advance the state of the art of intelligent computer-assisted instruction (ICAI) are the evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. Each issue is addressed via its theoretical context, key constructs, appropriate references to the literature, methodological aspects,…

Evaluation models are being developed for assessing artificial intelligence (AI) systems in terms of similar performance by groups of people. Natural language understanding and vision systems are the areas of concentration. In simplest terms, the goal is to norm a given natural language system's performance on a sample of people. The specific…

Some special problems associated with evaluating intelligent computer-assisted instruction (ICAI) programs are addressed. This paper intends to describe alternative approaches to the assessment and improvement of such applications and to provide examples of efforts undertaken and shortfalls. Issues discussed stem chiefly from the technical demands…