Previous research in second language writing has shown that when scoring performance assessments even trained raters can exhibit significant differences in severity. When raters disagree, using discussion to try to reach a consensus is one popular form of score resolution, particularly in contexts with limited resources, as it does not require…

The aim of this study is to examine the variability in and reliability of scores assigned to different quality EFL compositions by EFL instructors and their rating behaviors. Using a mixed research design, quantitative data were collected from EFL instructors' ratings of 30 compositions of three different qualities using a holistic scoring rubric.…

This study examined the effects of identity revelation and concealment on the number of times students' work was assessed in an online peer assessment context. It also examined the underlying reasons guiding the assessor's targeting behavior. Two fifth-grade classes participated. The one-group pretest-posttest experimental research design coupled…

The Objective Structured Clinical Exam (OSCE) is a widely used method of assessment in medical education. Rater cognition has become an important area of inquiry in the medical education assessment literature generally, and in the OSCE literature specifically, because of concerns about potential compromises of validity. In this study, a novel…

In this paper, the results of an investigation into the training needs and expectations of participants of professional development training for specialist teachers and assessors of learners with literacy difficulties and dyslexia are discussed and analysed. The methodology includes both qualitative and quantitative data which were collected over…

This paper reviews a selected sample of 24 doctoral dissertations in language assessment (broadly defined), completed between 2006 and 2011 in Canadian universities. These dissertations fall into five thematic categories: 1) reliability, validity and factors affecting test performance; 2) washback (impact) and ethics; 3) raters, rating and rating…

Purpose: This study aims to investigate quality of feedback as offered by supervisor-assessors with varying levels of assessor expertise following assessment of performance in residency training in a health care setting. It furthermore investigates if and how different levels of assessor expertise influence feedback characteristics.…

As part of a mixed-methods doctoral project, this study drew on an interview, bracketed by two surveys, with experienced Australian reviewers. It explored questions about the contributions of peer review, the need and possibility to improve it and the driving factors for reviewers to contribute voluntarily. The respondents emphasised the essential…

What it means to be a "professional" in further education (FE) in England has been the subject of ongoing debate over the last two decades. In an attempt to codify professionalism, New Labour developed a package of reforms, crystallised by the introduction of professional standards and qualifications and a new inspection framework under…

Along with a more humanitarian movement in language testing, accountability to contextual variables in the design and development of any assessment enterprise is emphasized. However, when it comes to writing assessment, it is found that multiplicity of rating scales developed to fit diverse contexts is mainly headed by well-known native testing…

This mixed-methods study examines the effects of rating scale length and rater experience on listeners' judgments of second-language (L2) speech. Twenty experienced and 20 novice raters, who were randomly assigned to 5-point or 9-point rating scale conditions, judged speech samples of 38 newcomers to Canada on numerical rating scales for…

Targeting Outcomes of Programs (TOP) is a seven-step hierarchical programming model in which the program development and performance sides are mirror images of each other. It served as a framework to identify a simple method for targeting photographic events in nonformal education programs, indicating why, when, and how photographs would be useful…

By centralizing the issue of test fairness in language proficiency assessments, this study responds to a call by researchers for developing greater social responsibility in the language testing agenda. As inquiries into language attitude and psychology indicate, there is an underlying uncertainty pertaining to the validity of test use and score…

This paper explores library evaluation in Australia and proposes a return to research fundamentals in which evaluators are asked to consider the centrality of philosophical issues and the role of different research methods. A critique of current evaluation examples demonstrates a system-centred, quantitative, input/output focus which fails to…