RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

Comparative judgment is gaining popularity as an assessment tool, including for high-stakes testing purposes, despite relatively little research on the use of the technique. Advocates claim two main rationales for its use: that comparative judgment is valid because humans are better at comparative than absolute judgment, and because it distils the…

As part of its "Guiding Principles for Evaluators," the American Evaluation Association (AEA) requires that evaluators develop cultural competencies. Using a successive-independent-samples design, the researchers sought to compare perceptions of cultural competence across a duration of 10 years. Qualitative data were collected via online…

Recent literature places more emphasis on assessment comments rather than relying solely on scores. Both are variable, however, emanating from assessment judgements. One established source of variability is "contrast effects": scores are shifted away from the depicted level of competence in a preceding encounter. The shift could arise…

A curriculum map is a visualisation of relationships within and between a curriculum or curricula. Curriculum mapping refers to the method for creating and using the curriculum map, however this term is used broadly and encompasses a variety of methodological approaches. Often, researchers in the field of curriculum studies conduct curriculum…

Culturally responsive evaluation contests that Western-derived evaluation methods represent a universal truth and promotes approaches that reflect the local context. Taking this approach, we examine how the method of contribution analysis may be modified to reflect a different worldview. We reframe contribution analysis using a Confucian lens as…

Email has become the most preferred form of business communication. Writing "good" email has become an essential skill required in the industry. "Good" email writing not only facilitates clear communication, but also makes a positive impression on the recipient, whether it be one's colleague or a customer. The aim of this paper…

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson, Hood, & Hughes, 2002; Kirkhart & Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu…

This article reports on the first Florida statewide assessment of the Gold Seal Quality Care program, accreditation, and the relationship with licensing violations. This study analyzed the differences between the Department of Children and Families Gold Seal-Accredited facilities and nonaccredited facilities by comparing the facilities and the…

This paper describes the profile of verbal response modes utilised in the expert application of Short-Term Dynamic Psychotherapy (STDP). One hundred and fifteen randomly selected segments from six treatments of STDP were analysed. Trained raters used a verbal response mode coding system to examine the individual speaking turns of an expert…

In the Swedish educational system, teachers have the dual responsibility of assigning final grades and marking their own students' national tests. The Government has mandated the Swedish Schools Inspectorate to remark samples of the national tests to see if teacher marking can be trusted. Reports from this project have concluded that intermarker…

This paper discusses one of the fundamental issues in education and social program evaluation: the proper role for an evaluator. Based on respective and comparative analysis of five theorists' positions on this fundamental issue, this paper reveals how different perspectives on other fundamental issues in evaluation such as value, methods, use and…

This paper describes one reviewer's experience of reviewing for the journal "System" over an eight-year period, 2003-2011. The paper reports on the reviews produced by the single reviewer, which have been compiled into a specific purpose--an "occluded"--corpus (Swales, 1996) of 122 reviews, comprising 93,000 words. The paper first describes the…

Riley's Stuttering Severity Instrument (SSI) is widely used. The manuals allow SSI assessments to be made in different ways (e.g. from digital recordings or whilst listening to speech live). Digital recordings allow segments to be selected and listened to, whereas the entire recording has to be judged when listened to live. Comparison was made…

We investigated the scoring of the Speaking section of the Test of English as a Foreign Language[TM] Internet-based (TOEFL iBT[R]) test by speakers of English and one or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the TOEFL examinees with mixed first languages…