I propose two practical advances to the argument-based approach to validity: developing a living document and incorporating preregistration. First, I present a potential structure for the living document that includes an up-to-date summary of the validity argument. As the validation process may span across multiple studies, the living document…

Most existing solutions to the current replication crisis in science address only the factors stemming from specific poor research practices. We introduce a novel mechanism that leverages the experts' predictive abilities to analyze the root causes of replication failures. It is backed by the principle that the most accurate predictor is the most…

While many instructors are aware of the "Literary Digest" 1936 poll as an example of biased sampling methods, this article details potential further explorations for the "Digest's" 1924-1936 quadrennial U.S. presidential election polls. Potential activities range from lessons in data acquisition, cleaning, and validation, to…

Think alouds are valuable tools for academicians, test developers, and practitioners as they provide a unique window into a respondent's thinking during an assessment. The purpose of this special issue is to highlight novel ways to use think alouds as a means to gather evidence about respondents' thinking. An intended outcome from this special…

The Student Evaluation of Teaching (SET) instrument provides insight for instructors and administrators alike, often touting high response-rates to endorse their validity and reliability. However, response-rate alone omits consideration for "adequate quantity of 'observational sampling opportunity' (OSO) data points" (e.g., high student…

In this article, we interrogate the concept of methodological 'failures' as they arise during fieldwork, in the process of collecting empirical data. We highlight how the techniques of validity horizon matrices and power analysis can be used as methodological tools to illustrate moments in the fieldwork process where these 'failures' occur and to…

This article examines the concept of credible evidence in Extension evaluations with specific attention to the measures and measurement strategies used to collect and create data. Credibility depends on multiple factors, including data quality and methodological rigor, characteristics of the stakeholder audience, stakeholder beliefs about the…

Across all sciences, the quality of measurements is important. Survey measurements are only appropriate for use when researchers have validity evidence within their particular context. Yet, this step is frequently skipped or is not reported in educational research. This article briefly reviews the aspects of validity that researchers should…

The purpose of this article is to move the field of intellectual and closely related developmental disabilities (IDD) towards a better understanding of evidence and evidence-based practices. To that end, we discuss (a) different perspectives on and levels of evidence, (b) commonly used evidence-gathering strategies, (c) standards to evaluate…

This article outlines Indigenous data collection as a methodology that expands a sampling frame by using tribal Indigenous cultural knowledge, such as creation stories like those of the Cocopah and Quechan nations, to address limitations found in the methodological quality of Native American survey data in currently available datasets. Indigenous…

This chapter proposes approaches for assessing learners and evaluating courses and curriculum that could be used by directors of health professions education (HPE) programs to determine the effectiveness and impact of their programs.

This paper addresses a relatively unexplored area in the field of learning analytics: how analytics are taken up and used as part of teaching and learning processes. Initial steps are taken towards developing design knowledge for this "middle space," with a focus on students as analytics users. First, a core set of challenges for…

There are usually limited user evaluation of resources on a recommender system, which caused an extremely sparse user rating matrix, and this greatly reduce the accuracy of personalized recommendation, especially for new users or new items. This paper presents a recommendation method based on rating prediction using causal association rules.…

The purpose of this paper is to introduce the process and value of Random Qualitative Validation (RQV) in the development and interpretation of survey data. RQV is a method of gathering clarifying qualitative data that improves the validity of the quantitative analysis. This paper is concerned with validity in relation to the participants'…

The purpose of this document is to serve as a reference guide for States involved in the P-20 effort to map data elements to the Common Education Data Standards (CEDS). It describes how one State used Align to assist in development of its P-20 system and describes in detail how this project could be replicated. The CEDS provide a shared vocabulary…