Classification evaluation metrics are often used to evaluate adaptive tutoring systems-- programs that teach and adapt to humans. Unfortunately, it is not clear how intuitive these metrics are for practitioners with little machine learning background. Moreover, our experiments suggest that existing convention for evaluating tutoring systems may…

Using exogenous characteristics to identify endogenous subgroups, the approach discussed in this method note creates symmetric subsets within treatment and control groups, allowing the analysis to take advantage of an experimental design. In order to maintain treatment--control symmetry, however, prior work has posited that it is necessary to use…

This study addresses the central question "How do evaluators' background characteristics relate to their evaluation design choices?" Evaluators were provided with a fictitious description of a school-based program and asked to design an evaluation of that program. Relevant background characteristics such as level of experience,…

A simulation study was conducted in an attempt to examine how evaluators modify their evaluation design in response to differing stakeholder groups. In this study, evaluators were provided with a fictitious description of a school-based program. They were then asked to design an evaluation of the program. After the evaluation design decisions were…

Considerable research has been conducted on the use of the findings of program evaluation, but little, if any, attention has been paid to the soundness of the methods of this research. If the methods are not sound or not well described in the research, the strength of the conclusions of the research is unknown. The authors examine the empirical…

This paper presents the usability evaluation of a digital systems laboratory class offered to distance-learning students. It details the way in which students can participate remotely in such a laboratory, the methodology employed in the usability assessment of the laboratory infrastructure (hardware and software), and also outlines the main…

This report describes work using nationally representative 2005 data from the School Nutrition Dietary Assessment-III (SNDA-III) study to develop a simulation model to predict the potential implications of changes in policies or practices related to school meals and school food environments. The model focuses on three domains of outcomes: (1) the…

This project, located in a large metropolitan school district, concerns the operation and evaluation of three Mathematics Demonstration Centers. In the first phase, directions are given about the necessary information to be gathered by the evaluating organization, (e.g. the school district administrative organization, size of school district,…

Describes an evaluation of the effectiveness of simulations developed to teach and assess nurses' clinical decision-making skills. Includes recommendations for using this method of evaluation and for improving nurses' judgment skills. (Author)

Shows how one can combine a large-scale planned variation experimental design and multilevel analysis to address research questions that go beyond the issue of overall treatment effectiveness. Uses data from a simulated field experiment to describe the multilevel models that can be used. (SLD)

The preparation of this simulation material package is guided by the concept of an evaluator as a decision-maker, based on the definition of evaluation as a continuous assessment concerned with answering decision-making questions. The continuous concept of evaluation is based on the model created by Egon Guba and Daniel Stufflebeam, named by its…

Types of assessment, five assessment techniques, and issues related to assessing experiential learning in career education are discussed. The following six points should be considered before an assessment program begins: (1) assumptions; (2) goals; (3) outcomes; (4) learning environment; (5) role of assessors; and (6) focus. Two basic types of…

DETECT, the acronym for Dimensionality Evaluation To Enumerate Contributing Traits, is an innovative and relatively new nonparametric dimensionality assessment procedure used to identify mutually exclusive, dimensionally homogeneous clusters of items using a genetic algorithm ( Zhang & Stout, 1999). Because the clusters of items are mutually…

The CEVAL simulation exercise is appropriate for use in a workshop setting with up to 30 persons. It gives non-administrative groups--school board members, parent advisory councils, teachers, and parents--experience in making evaluative decisions, from the perspective of a school principal evaluating five special programs in an elementary school.…

As change occurs in various health sciences programs, evaluational strategies should be developed so that adaptive decisions may be made. Evaluation models taking into account inputs, methodology, and outputs (Stake, 1967, and Astin and Panes, 1971) need examination. Alternative measurement instruments for formative and summative evaluations,…