The impact of misspecifying covariance matrices at the second and third levels of the three-level model is evaluated. Results indicate that ignoring existing covariance has no effect on the treatment effect estimate. In addition, the between-case variance estimates are unbiased when covariance is either modeled or ignored. If the research interest…

Inter-rater reliability, commonly assessed by intra-class correlation coefficient ICC, is an important index for describing the extent to which there is consistency amongst two or more raters in assigned measures. In organizational research, the data structure is often hierarchical and designs deviate substantially from the ideal of a balanced…

The purpose of this paper is to showcase new research that seeks to provide guidance on the heterogeneity of treatment effects by utilizing the variance of demographic differences in state assessments. This study is focused on a simple randomized block design where students are nested within schools, and within each school students are randomized…

A new design-based theory has recently been developed to estimate impacts for randomized controlled trials (RCTs) and basic quasi-experimental designs (QEDs) for a wide range of designs used in social policy research (Imbens & Rubin, 2015; Schochet, 2016). These methods use the potential outcomes framework and known features of study designs…

Aware of the challenges set before the Houston Independent school District by rapid growth in the numbers of English learner students, and a critical shortage of teachers with bilingual certification for more than a decade, members of Regional Educational Laboratory Southwest's English Learners Research Alliance sought information that districts…

These are the appendixes for the report "Children's Knowledge and Skills at Kindergarten Entry in Illinois: Results from the First Statewide Administration of the Kindergarten Individual Development Survey." At least half of states administer or are developing kindergarten entry assessments. In fall 2017 the Illinois State Board of…

These appendices accompany the report "Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18." Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the…

Background: Cluster randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels.…

The purpose of this study was to determine if validity coefficients for ACT scores and high school grade point average (HSGPA) decayed or held stable over eight semesters of undergraduate study in science, technology, engineering, and mathematics (STEM) fields at civilian four-year institutions, and whether the decay patterns differed from those…

The capacity of the field to conduct power analyses for group randomized trials (GRTs) of educational interventions has improved over the past decade (Authors, 2009). However, a power analysis depends on estimates of design parameters. Hence it is critical to build the empirical base of design parameters for GRTs across a variety of outcomes and…

Magnet schools are one of the largest sectors of choice schools in the United States. In this study, we explored whether there is heterogeneity in magnet school effects on student achievement by examining the effectiveness of 24 recently funded magnet schools in 5 school districts across 4 states. We used a two-step analysis: First, separate…

The current study examined the role of AP® Exam participation and performance on four-year college graduation in four years. Because students who take AP Exams can earn college credit while still in high school, it was expected that AP students would have higher four-year graduation rates. Moreover, it was expected that AP students who earned…

Achieve3000 publishes a number of online literacy programs that differentiate lessons and activities based on student performance. Magnolia Consulting--an independent, third party evaluation firm--assessed Achieve3000 Solutions' impact on student learning during the 2014-15 school year. The researchers used a randomized control trial, where…

In Belize, the percentage of untrained teachers (i.e. teachers with the equivalent of a high school education or less) is one of the highest in the region of Latin America and the Caribbean. In response to growing student cohorts, the increased demand for more classroom teachers has led to significant decrease in the proportion of trained teachers…

Numerous authors have reviewed research to determine which skills are considered fundamental to successful academic and social outcomes for students (Hattie, Briggs, & Purdie, 1996; Masten & Coatworth, 1998; Wang, Haertel, & Walberg, 1994; Zins, Weissberg, Wang, & Walberg, 2004). These skills include: (1) cognitive and…