The author provides information about reliability and areas that educators should examine in determining if an assessment is consistent and trustworthy for use, and how it should be interpreted in making decisions about students. Reliability areas that are discussed in the column include internal consistency, test-retest or stability, inter-scorer…

In this article, the authors share their reflections on how different experiences and paradigms have broadened their understanding of the work of assessment in higher education. As they collaborated to create a panel for the 2024 International Conference on Assessing Quality in Higher Education, they recognized that they, as assessment…

Multidimensional measuring instruments are often used in behavioral, social, educational, marketing, and biomedical research. For these scales, the paper discusses how to find the optimal score based on their components that is associated with the highest possible reliability. Within the framework of structural equation modeling, an approach to…

Traditional authentic assessment tasks are frequently tied to future work and enmeshed in neoliberal and capitalist visions of education. We advocate an alternative approach where authenticity signifies meaningful learning outside the confines of the classroom to promote deep learning that 'sticks'. We proffer an understanding of "assessment…

The remarkable strides in artificial intelligence (AI), exemplified by ChatGPT, have unveiled a wealth of opportunities and challenges in assessment. Applying cutting-edge large language models (LLMs) and generative AI to assessment holds great promise in boosting efficiency, mitigating bias, and facilitating customized evaluations. Conversely,…

The product Bayes factor (PBF) synthesizes evidence for an informative hypothesis across heterogeneous replication studies. It can be used when fixed- or random effects meta-analysis fall short. For example, when effect sizes are incomparable and cannot be pooled, or when studies diverge significantly in the populations, study designs, and…

Decision consistency (DC) is the reliability of a classification decision based on a test score. In professional credentialing, the decision is often a high-stakes pass/fail decision. The current methods for estimating DC are computationally complex. The purpose of this research is to provide a computationally and conceptually simple method for…

Practitioners often struggle to assess reflective learning in the workplace because of difficulties conceptualizing reflection and its effects in the workplace. This article addresses this problem by offering a pragmatic approach to assessment that asks practitioners to specify why they are using reflection, what they are hoping to gain from it,…

School- and college-based vocational and technical qualifications (VTQs) in England are required to award successful candidates a grade rather than simple pass or fail. Ensuring the reliability and validity of these grades is considered vital, particularly in light of the high-stakes purposes for which school assessment results in England are…

To identify the dominant cultural models among parents transmitting faith to their children, we find few methodological guidelines to guide coding and analysis of semi-structured interviews. We thus developed a three-phase procedure for our research team. Phase-one follows Campbell et al. by unitizing on meanings rather than words/pages, including…

The "Program Administration Scale" (PAS) is designed to reliably measure and improve the leadership and management practices of center-based programs--the only instrument of its kind to focus exclusively on organization-wide administrative issues. In the third edition, the authors share updated information supporting the reliability and…

Rater-mediated assessments are a common methodology for measuring persons, investigating rater behavior, and/or defining latent constructs. The purpose of this article is to provide a pedagogical framework for examining rater variability in the context of rater-mediated assessments using three distinct models. The first model is the observation…

Design is core element of general technology education internationally. While there is a degree of contention with regards to its treatment, there is general consensus that the inclusion of design in some form is important, if not characteristic, of the subject area. Acknowledging that design is important, there are many questions which need to be…

In 2021, the Council for Higher Education Accreditation (CHEA) made a monumental move to require postsecondary institutions to evaluate and document their actions to ensure fairness in admissions, an inclusive learning environment, and equitable student outcomes. Around the same time, a team comprising educational measurement experts, diversity…

Precision is a key facet of test development, with score reliability determined primarily according to the types of error one wants to approximate and demonstrate. This article identifies and discusses several primary forms of reliability estimation: internal consistency (i.e., split-half, KR-20, a), test-retest, alternate forms, interscorer, and…