Background: For decades, education researchers have relied on the work of Campbell, Cook, and Shadish to help guide their thinking about valid impact estimates in the social sciences (Campbell & Stanley, 1963; Shadish et al., 2002). The foundation of this work is the "validity typology" and its associated "threats to…

Context: Instrument validation is an important facet of survey research methods and athletic trainers must be aware of the important underlying principles. Objective: To discuss the process of survey development and validation, specifically the process of construct validation. Background: Athletic training researchers frequently employ the use of…

Monte Carlo methods can be used in data analytic situations (e.g., validity studies) to make decisions about sample size and to estimate power. The purpose of using Monte Carlo methods in a validity study is to improve the methodological approach within a study where the primary focus is on construct validity issues and not on advancing…

The purpose of this article is to demonstrate application of a new framework, the validation framework (VF), to assist researchers in evaluating mixed research studies. Based on an earlier work by Dellinger and Leech, a description of the VF is delineated. Using the VF, three studies from education, health care, and counseling fields are…

Recent Monte Carlo research has illustrated that the traditional method for assessing the construct-related validity of assessment center (AC) post-exercise dimension ratings (PEDRs), an application of confirmatory factor analysis (CFA) to a multitrait-multimethod matrix, produces inconsistent results [Lance, C. E., Woehr, D. J., & Meade, A. W.…

Structured questionnaires and semi-structured interviews are often used in mixed method studies to generate confirmatory results despite differences in methods of data collection, analysis, and interpretation. A review of 19 questionnaire-interview comparison studies found that consensus and consistency statistics were generally weak between…

The analysis of convergent and discriminant validity is an integral part of the construct validation process. Models for analyzing the convergent and discriminant validity have typically been developed for cross-sectional data. There exist, however, only a few approaches for longitudinal data that can be applied for analyzing the construct…

Hilton (2006) criticises the PIRLS (Progress in International Reading Literacy Study) tests and the survey conduct, raising questions about the validity of international surveys of reading. Her criticisms fall into four broad areas: cultural validity, methodological issues, construct validity and the survey in England. However, her criticisms are…

The primary purpose of this article is to further discussions of validity in mixed methods research by introducing a validation framework to guide thinking about validity in this area. To justify the use of this framework, the authors discuss traditional terminology and validity criteria for quantitative and qualitative research, as well as…

The study reported in this paper, a Multistage Longitudinal Comparative (MLC) Design Stage II evaluation conducted as a planned preliminary efficacy evaluation (psychometric evaluation of measures, short-term controlled outcome studies, etc.) of the Changing Lives Program (CLP), provided evidence for the reliability and validity of qualitative…

A theory-driven approach to validity is proposed. The central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. (Author/LMO)

This paper examines issues arising from the use of self-report questionnaires in cross-cultural contexts. The research draws from the extensive literature on cross-cultural leadership in business organizational culture as well as from educational cross-cultural contexts. It examines claims, drawn from business and educational contexts, that many…

This paper focuses on issues of data trustworthiness in praxis-oriented empirical work, research openly committed to the building of a world in which we can all flourish. The central argument is that those exploring the possibilities for a change-enhancing advocacy paradigm for doing empirical research in the human sciences must begin to be more…

The Rasch method, so often used in educational research, was used to analyze the construct validity of the criteria for premenstrual dysphoric disorder from "The Diagnostic and Statistical Manual of Mental Disorders." The method was used in contrast to using an external validation to establish the diagnostic utility of tests for…

The central thesis of this article is that the process of eliminating validity threats depends fundamentally on no-difference findings, a fact that has not been made explicit by researchers. The implications of this neglect are explored using examples from a number of different substantive areas such as psychology, health, and medicine.…