The issue of model uncertainty has been gaining interest in education and the social sciences community over the years, and the dominant methods for handling model uncertainty are based on Bayesian inference, particularly, Bayesian model averaging. However, Bayesian model averaging assumes that the true data-generating model is within the…

Logistic regression (LR)-based methods have become increasingly popular for predicting and articulating cut scores. However, the precision of predictive relationships is largely dependent on the underlying correlations between the predictor and the criterion. In two simulation studies, we evaluated the impact of varying the underlying grade-level…

Using an empirically-based simulation study, we show that typically used methods of choosing an item calibration sample have significant impacts on achievement bias and system rankings. We examine whether recent PISA accommodations, especially for lower performing participants, can mitigate some of this bias. Our findings indicate that standard…

We investigate whether commonly used value-added estimation strategies produce accurate estimates of teacher effects under a variety of scenarios. We estimate teacher effects in simulated student achievement data sets that mimic plausible types of student grouping and teacher assignment scenarios. We find that no one method accurately captures…

Lean is regarded as a systematic approach to maximizing value by minimizing waste, and by flowing the product or service at the pull of the customer demand. These key concepts of "value," "flow," and "pull," align with the ultimate lean goal: "perfection," or a continuous striving for improvement in the…

This dissertation comprises three papers that propose, discuss, and illustrate models to make improved inferences about research questions regarding student achievement in education. Addressing the types of questions common in educational research today requires three different "extensions" to traditional educational assessment: (1)…

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However,…

Value-added modeling is an alternative approach to test-based accountability systems based on the proportions of students scoring at or above pre-determined proficiency levels. Value-added modeling techniques provide opportunities to estimate an individual teacher's effect on student learning, while allowing for the possibility to control for the…

In this paper, we present our experiences in using two automatic assessment tools, TRAKLA and TRAKLA2, in a second course of programming. In this course, 500-700 students have been enrolled annually during the period 1993-2004. The tools are specifically designed for assessing algorithm simulation exercises in which students simulate the working…

A set of constructed portfolios containing achievement products and background information for a simulated student was used to investigate how teachers formulate the evaluation of achievement of their students. The materials, developed for a simulated student named "Chris," were presented to 147 student teachers who graded the 8 product…

Discussion of assessing student performance on simulation exercises focuses on a study that examined the relationship between financial performance and various other measures of student learning in a business simulation. The use of Bloom's taxonomy of learning is explained and further research is suggested. (14 references) (LRW)

The Formative Evaluation and Heuristic Research (FEHR) Practicum, a computer-simulated educational research and evaluation experience, was assessed as a means of improving participants' knowledge and skill in traditional research. Trainee activities include the preparation of preliminary surveys, evaluation proposals, budgets, computer commands,…

Several new developments in educational measurement which contribute to a broadening of assessment options were presented at the conference on Alternative Conceptions of Competence. A common theme synthesizing these alternatives was a view of the complexity of human competence and the inadequacy of the multiple-choice examination as a measure of…

A study compared the performance of 232 Brown University (Rhode Island) medical graduates on senior-year examinations in clinical skills examinations and during residencies, as evaluated by their residency directors. Results indicated a strong correlation between the two evaluations, suggesting that the performance-based undergraduate assessments,…

The failure to find strong instructional effects of simulation is usually attributed to poor measurement and faulty research design. However, weak implementation of simulation into the classroom, rather than methodological inadequacy, is presented here as the primary reason for finding insignificant effects. The types of environments under which…