This paper addresses instrumentation design and an ongoing study of an online formative assessment instrument and its effects in pre-college (developmental/remedial) mathematics courses. The use of facet theory and instructional technologies are harmonized to construct and scale an online instrument designed to document student procedures while…

Because of the importance of beliefs, mathematics educators need to consider ways to assess beliefs and belief change. Beliefs, because they must be inferred, can be difficult to measure, particularly with a common metric that enables one to compare individuals. Some limitations of Likert scales are identified and overcome in a newly developed…

Large-scale testing programs are generally based on the assumptions that the test-takers experience standard conditions for taking the test and that everyone will do his or her own work without having prior knowledge of specific questions. These assumptions are not necessarily true. The ways students and educators use to get around standardizing…

This report examines issues in the scale-up of assessment software from the Center for Research on Evaluation, Standards, and Student Testing (CRESST). "Scale-up" is used in a metaphorical sense, meaning adding new assessment tools to CRESST's assessment software. During the past several years, CRESST has been developing and evaluating a…

Procedures used to establish the comparability of scores derived from the College Board Admissions Testing Program (ATP) computer adaptive Scholastic Aptitude Test (SAT) prototype and the paper-and-pencil SAT are described in this report. Both the prototype, which is made up of Verbal and Mathematics computer adaptive tests (CATs), and a form of…

The purpose of this study was to determine if a relationship exists between scores and the times that medical students choose to take a computer-administered test. The results indicate that students who choose to take a test later within a given time period tend to perform less well than students who take the test earlier. Although the magnitude…

The purpose of this study was to compare and evaluate five online pretest item calibration/scaling methods in computerized adaptive testing (CAT): (1) the marginal maximum likelihood estimate with one-EM cycle (OEM); (2) the marginal maximum likelihood estimate with multiple EM cycles (MEM); (3) Stocking's Method A (M. Stocking, 1988); (4)…

An essay test is now an integral part of the computer based Test of English as a Foreign Language (TOEFL-CBT). This paper provides a brief overview of the current TOEFL-CBT essay test, describes the operational procedures for essay scoring, including the Online Scoring Network (OSN) of the Educational Testing Service (ETS), and discusses major…

This paper provides a technology-based review of World Wide Web-based testing technologies. It suggests an evaluation framework that could be used by practitioners in Web-based education to understand and compare features available in various Web-based testing systems. In order to compare existing options, the life cycle of a question in Web-based…

Two operational item pools were used to investigate the performance of stratum computerized adaptive tests (CATs) when items were assigned to strata based on empirical estimates of item difficulty or human judgments of item difficulty. Items from the first data set consisted of 54 5-option multiple choice items from a form of the ACT mathematics…

Error indices (bias, standard error of estimation, and root mean square error) obtained on different scales of measurement under different test termination rules in a computerized adaptive test (CAT) context were examined. Four ability estimation methods were studied: (1) maximum likelihood estimation (MLE); (2) weighted likelihood estimation…

Setting standards on educational tests is extremely challenging. The psychometric literature is replete with methods and guidelines for setting standards on educational tests; however, little attention has been paid to the process of setting standards on computerized adaptive tests (CATs). This lack of attention is unfortunate because CATs are…

Standard setting, when carefully done, can be an expensive and time-consuming process. The modified Angoff method and the benchmark method, as utilized in this study, employ representative panels of judges to provide recommended passing scores to standard setting decision-makers. It has been considered preferable to have the judges meet in a…

Two experiments reproduced in a simulated computerized test-taking situation the effect of two of the main determinants in answering an item in a test: the difficulty of the item and the time available to answer it. A model is proposed for the time to respond or abandon an item and for the probability of abandoning it or answering it correctly. In…

Computerized adaptive testing allows us to create a unique personalized test that matches the ability and knowledge of the test taker. (Author)