"Integrative evaluation" is an approach with two main phases: identification of plausible rival hypotheses and integration of rival hypotheses. The first phase may correspond to traditional adversary evaluation, whereas the second phase, that is not included in adversary evaluation, requires integrative thinking which can be applied when…

Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational Progress (NAEP) data mining competition. The…

This study investigated the hypothesis that cognitive performance of students with physical disabilities may be influenced by the evaluators' identity. Students with or without a physical disability completed a logic test and were informed that they would be evaluated by students from their own group (ingroup condition) or from an other group…

To begin statistical analysis, Bayesians quantify their confidence in modeling hypotheses with priors. A prior describes the probability of a certain modeling hypothesis apart from the data. Bayesians should be able to defend their choice of prior to a skeptical audience. Collaboration between evaluators and stakeholders could make their choices…

In this study, data on grades awarded for bachelor and master theses at a large Polish university were used to identify possible discrimination on gender or physical attractiveness. The focus is on the gap between the grades awarded by the advisor (who knows the student personally) and the referee (who typically does not, so that gender is less…

Contradictory findings from "well-implemented" rigorous evaluations invite researchers to identify the differences that might explain the contradictions, helping to generate testable hypotheses for new research. This panel will examine efforts to ensure that the large number of local evaluations being conducted as part of four…

If Christian schools desire students to achieve higher-level thinking, then the textbooks that teachers use should reflect such thinking. Using Risner's (1987) methodology, raters classified questions from two Christian publishers' fifth grade reading textbooks based on the revised Bloom's taxonomy (Anderson et al., 2001). The questions in the A…

Assessing the comparability of different groups is an issue facing many researchers and evaluators in a variety of settings. Commonly, null hypothesis significance testing (NHST) is incorrectly used to demonstrate comparability when a non-significant result is found. This is problematic because a failure to find a difference between groups is not…

Usage-based theories of language learning suggest that native speakers of a language are acutely aware of formulaic language due in large part to frequency effects. Corpora and data-driven learning can offer useful insights into frequent patterns of naturally occurring language to second/foreign language learners who, unlike native speakers, are…

The experiment reported in this article investigated the assumption that visual movement plays a role in musician-to-audience communication in marimba performance. Body movement is of particular relevance here as the expressive capabilities of the marimba are relatively restricted, and the movements required to play it are visible. Twenty-four…

Hypothesizing that prejudiced feelings about minorities are due not only to racial differences but to the fact that minorities often have lower socio-economic rank as well, this study examines data from a household survey in Tulsa, Oklahoma and documents the greater salience of socioeconomic cues and the lesser salience of racial cues for social…

Causes of evaluator bias are: overemphasizing concrete, salient, and retrievable information; reporting only evidence which confirms hypothesis; focusing on stable personality factors, rather than on situation and environment; developing positive perceptions of a program as both an evaluator and a highly involved participant; statistical naivete;…

Hypothesis generation and testing is outlined as an additional domain for program evaluators. Program evaluation involves a thorough analysis of the processes that contribute to change (or a lack of change) among program recipients. This process of change is analyzed in two ways: (1) treating programs as naturally occurring field studies; and (2)…

In spite of many inherent problems, impact evaluation should remain an integral part of program evaluation, both because impact evaluation brings its budget justification with it, and because its focus on causal hypotheses is an essential part of evaluation. Methodology based on the author's body of work is reviewed. (SLD)