Human ratings are ubiquitous in creativity research. Yet, the process of rating responses to creativity tasks -- typically several hundred or thousands of responses, per rater -- is often time-consuming and expensive. Planned missing data designs, where raters only rate a subset of the total number of responses, have been recently proposed as one…

Who should evaluate the originality and task-appropriateness of a given idea has been a perennial debate among psychologists of creativity. Here, we argue that the most relevant evaluator of a given idea depends crucially on the level of expertise of the person who generated it. To build this argument, we draw on two complimentary theoretical…

The literature on Automated Essay Scoring (AES) systems has provided useful validation frameworks for any assessment that includes AES scoring. Furthermore, evidence for the scoring fidelity of AES systems is accumulating. Yet questions remain when appraising the scoring performance of AES systems. These questions include: (a) which essays are…

In this research report, we describe the design and empirical findings for a large-scale study of essay writing ability with approximately 2,500 high school students in Germany and Switzerland on the basis of 2 tasks with 2 associated prompts, each from a standardized writing assessment whose scoring involved both human and automated components.…

Archived transcripts from tens of millions of online human tutoring sessions potentially contain important knowledge about how online tutors help, or fail to help, students learn. However, without ways of automatically analyzing these large corpora, any knowledge in this data will remain buried. One way to approach this issue is to train an…

A general approach to the analysis of nominal-scale ratings is discussed that is based on a simple measurement error model for a rater's judgments. The basic measurement error model gives rise to an agreement model for the agreement matrix of two or more raters. (SLD)

Program theory and implementation process theory contribute, respectively, to specifying the what and the why of program delivery. This article discusses the relationship between these two theories and the types of data collection methods that can be derived from each in order to contribute to more comprehensive evaluation designs. (JAZ)

In reacting to five recent books on program evaluation, eight evaluation perils are discussed: poor preparation of evaluations, programs with unrealistic goals, the mix-up of models, interpersonal conflicts, the morass of measurement, answering evaluation questions, the secondary evaluator as sniper, and hostility from program staff. (CM)

This paper addresses the main issues related to concepts of school effectiveness and their implications, with a focus on the selection of school-performance indicators. School effectiveness has many definitions, which is in part due to the different demands made by various social groups. The boundaries of performance within which schools should…