When and how to survey potential respondents is often determined by budgetary and external constraints, but choice of survey modality may have enormous implications for data quality. Different survey modalities may be differentially susceptible to measurement error attributable to interviewer assignment, known as interviewer effects. In this…

More aggregate data on school performance is available than ever before, opening up new possibilities for applied researchers interested in assessing the effectiveness of school-level interventions quickly and at a relatively low cost by implementing comparative interrupted times series (CITS) designs. We examine the extent to which effect…

The purpose of this paper is to introduce the Five-Level Qualitative Data Analysis (5LQDA) method for ATLAS.ti as a way to intentionally design methodological approaches applicable to the field of evaluation. To demonstrate my analytical process using ATLAS.ti, I use examples from an existing evaluation of a STEM Peer Learning Assistant program.…

Engaging students in assessment practice benefits honors students, faculty, and administrators. Students gain meaningful research experience while honors programs receive data to help assess student learning and prepare for program review. A one-semester course, Program Evaluation Experiences, tasks students (n = 10) with collecting and analyzing…

Intended to guide scholars who are new to qualitative research, this methods brief focuses mostly on what reviewers look for in manuscripts submitted for publication. The author acknowledges that reviewers' preferences likely reflect their theoretical perspectives, research-oriented coursework, mentors, and research and writing experiences. The…

This paper reports on the external evaluation of a two-year project aimed at improving educational opportunities for Aboriginal Education Workers (AEWs) through technology-based pedagogy. A design-based approach was selected by the research team to increase the relevance of the research for both praxis and practice, with emphasis on improvement of…

Programs that improve teaching effectiveness represent a core strategy to improve student educational outcomes and close student achievement gaps. This article compiles empirical values of intraclass correlations for designing effective and efficient experimental studies evaluating the effects of these programs. The Early Childhood Longitudinal…

Purpose: The purposes of this paper are to provide a description of AI and to document and compare two applications of AI, one in program evaluation and another in an applied research study. Design/methodology/approach: Focus groups, interviews and observations were used to gather rich qualitative data which was used to detail Appreciative…

This article provides insight into how an adequate approach to selecting methods can establish credible and actionable evidence. The authors offer strategies to effectively support Extension professionals, including program developers and evaluators, in being more deliberate when selecting appropriate qualitative and quantitative methods. In…

The present study aims to examine the graduate studies related to in-service training programs for teachers conducted between the years 2000 and 2018 in Turkey. In this context, the content analysis of 88 graduate studies, including 71 master thesis and 17 dissertations, was conducted for the in-service teacher training accessible to the Council…

This response aims to critically evaluate the methodology and aims of the meta-analytic review written by Maggin and Johnson (2014). The present authors systematically provide responses for each of the original criticisms and highlight concerns regarding Maggin and Johnson's methodology, while objectively describing the current state of evidence…

This chapter provides examples of innovative methods and tools to collect, analyze, and report both quantitative and qualitative data in student affairs assessment.

Short comparative interrupted times series (CITS) designs are increasingly being used in education research to assess the effectiveness of school-level interventions. These designs can be implemented relatively inexpensively, often drawing on publicly available data on aggregate school performance. However, the validity of this approach hinges on…

This article presents a content analysis of the 2013 School Leadership Program (SLP) grants. SLP projects provide a unique opportunity for participants in the field to explore innovative leadership preparation and development and their impact on program participants, schools, school districts, and students. The article begins with an overview of…

The growth in the availability of longitudinal data--data collected over time on the same individuals--as part of program evaluations has opened up exciting possibilities for evaluators to ask more nuanced questions about how individuals' outcomes change over time. However, in order to leverage longitudinal data to glean these important insights,…