The number of performance assessments continues to increase around the world, and it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study describes an unfolding model for examining rater accuracy. Accuracy is defined as the difference between observed and expert ratings. Dichotomous accuracy…

This study examined the relative effectiveness of Bayesian model comparison methods in selecting an appropriate graded response (GR) model for performance assessment applications. Three popular methods were considered: deviance information criterion (DIC), conditional predictive ordinate (CPO), and posterior predictive model checking (PPMC). Using…

In the social sciences, latent traits often have a hierarchical structure, and data can be sampled from multiple levels. Both hierarchical latent traits and multilevel data can occur simultaneously. In this study, we developed a general class of item response theory models to accommodate both hierarchical latent traits and multilevel data. The…

A methodology is proposed to optimize the training classification performance of any suboptimal model. The method, referred to as univariate optimal discriminant analysis (UniODA), is illustrated through application to a two-group logistic regression analysis with 12 empirical examples. Maximizing percentage accuracy in classification is…

The generalizability of student evaluations of college instruction is analyzed and generalizability theory data collected in a common design is applied. Results suggest that use of student evaluations of teacher effectiveness may lead to decisions based upon undependable data. (Author/GK)

Results from a performance assessment in which 68 high school students wrote essays support the use of latent variable modeling for estimating reliability, concurrent validity, and generalizability of a scoring rubric. The latent variable modeling approach overcomes the limitations of certain conventional statistical techniques in handling…