The author provides information about reliability and areas that educators should examine in determining if an assessment is consistent and trustworthy for use, and how it should be interpreted in making decisions about students. Reliability areas that are discussed in the column include internal consistency, test-retest or stability, inter-scorer…

In this article, the authors share their reflections on how different experiences and paradigms have broadened their understanding of the work of assessment in higher education. As they collaborated to create a panel for the 2024 International Conference on Assessing Quality in Higher Education, they recognized that they, as assessment…

Practitioners often struggle to assess reflective learning in the workplace because of difficulties conceptualizing reflection and its effects in the workplace. This article addresses this problem by offering a pragmatic approach to assessment that asks practitioners to specify why they are using reflection, what they are hoping to gain from it,…

Rater-mediated assessments are a common methodology for measuring persons, investigating rater behavior, and/or defining latent constructs. The purpose of this article is to provide a pedagogical framework for examining rater variability in the context of rater-mediated assessments using three distinct models. The first model is the observation…

Comparative Judgement (CJ) has emerged as a technique that typically makes use of holistic judgement to assess difficult-to-specify constructs such as production (speaking and writing) in Modern Foreign Languages (MFL). In traditional approaches, markers assess candidates' work one-by-one in an absolute manner, assigning scores to different…

Precision is a key facet of test development, with score reliability determined primarily according to the types of error one wants to approximate and demonstrate. This article identifies and discusses several primary forms of reliability estimation: internal consistency (i.e., split-half, KR-20, a), test-retest, alternate forms, interscorer, and…

Like instruments used in afterschool programs to assess children's social and emotional growth or to evaluate staff members' performance, instruments used to evaluate program quality should be free from bias. Practitioners and researchers alike want to know that assessment instruments, whatever their type or intent, treat all people fairly and do…

In Gargani and Strong (2014), we describe The Rapid Assessment of Teacher Effectiveness (RATE), a new teacher evaluation instrument. Our account of the validation research associated with RATE inspired a review by Good and Lavigne (2015). Here, we reply to the main points of their review. We elaborate on the validity, reliability, theoretical…

The Autism Mental Status Examination (AMSE) described here is an eight-item observational assessment that prompts the observation and recording of signs and symptoms of autism spectrum disorders (ASD). The AMSE is intended to take place seamlessly in the context of a clinical exam and produces a total score. Subjects were independently…

Objective: Although health professionals increasingly are expected to be able to assess and manage patients' risk for suicide, few methods are available to evaluate this competency. This report describes development of a competency-assessment instrument for suicide risk-assessment (CAI-S), and evaluates its use in an objective structured clinical…

This paper introduces the Individualized Classroom Assessment Scoring System (inCLASS), an observation tool that targets children's interactions in preschool classrooms with teachers, peers, and tasks. In particular, initial evidence is reported of the extent to which the inCLASS meets the following psychometric criteria: inter-rater reliability,…

At the urging of the U.S. Department of Education, regional accrediting organizations have, during the last few years, placed much greater emphasis on implementation of assessment as a requirement for accreditation. Assessment serves two important purposes: institutional survival and improvement of student achievement. Of these two, survival is…

To find out whether the education community shares a collective understanding about how students should be evaluated, we surveyed 202 educators (from all grade levels) and scientists attending assessment workshops (Pennsylvania, California, and Massachusetts) or judging a national student competition (Washington, DC). The educators and scientists…

This article reports and demonstrates two SPSS macros for calculating Krippendorff's alpha and intraclass reliability coefficients in repetitive situations where numerous coefficients are needed. Specifically, the reported SPSS macros were used to evaluate the interrater agreement and reliability of student evaluations of teaching in thousands of…

Portfolio assessment has become the predominant best practice in writing assessment at the college level. Despite its clear superiority to previous methods of assessment, portfolio assessment has brought its own collection of challenges. Although the stated goal of much portfolio grading is to create an overall, or holistic, judgment of a…