Educators commonly use school climate survey scores to inform and evaluate interventions for equitably improving learning and reducing educational disparities. Unfortunately, validity evidence to support these (and other) score uses often falls short. In response, Whitehouse et al. proposed a collaborative, two-part validity testing framework for…

Hierarchical linear modeling (HLM) has become an increasingly popular multilevel method of analyzing data among nested datasets, in particular, the effect of specialized academic programming within schools. The purpose of this methodological study is to demonstrate the use of HLM to determine the effectiveness of STEM programming in an Ohio middle…

Measuring and understanding the nature of informant/rater effects and differences (Level 1) on a common trait when the target of measurement is at the organizational level (Level 2) involves a number of methodological considerations. Although previous research has discussed single-level latent variable applications of the correlated…

Organizational relationships are complex. Employees do their work as individuals, but also as members of larger teams. They exist within various social networks, both within and spanning organizations. Multilevel theory is at the core of the organizational sciences, and unpacking multilevel relationships is fundamental to the challenges faced…

Valid inferences on teaching drawn from students' test scores require that tests are sensitive to the instruction students received in class. Accordingly, measures of the test items' instructional sensitivity provide empirical support for validity claims about inferences on instruction. In the present study, we first introduce the concepts of…

K-12 intervention studies often include fidelity of implementation (FOI) as a mediating variable, though most do not report the validity of fidelity measures. This article discusses the critical need for validated FOI scales. To illustrate our point, we describe the development and validation of the Implementation Validity Checklist (IVC-R), an…

The selection of an appropriate booklet design is an important element of large-scale assessments of student achievement. Two design properties that are typically optimized are the "balance" with respect to the positions the items are presented and with respect to the mutual occurrence of pairs of items in the same booklet. The purpose…

The current study applied the person-fit statistic, l[subscript z], to data from a Canadian provincial achievement test to explore the usefulness of conducting person-fit analysis on large-scale assessments. Item parameter estimates were compared before and after the misfitting student responses, as identified by l[subscript z], were removed. The…

This study examines the use of cross-classified random effects models (CCrem) and cross-classified multiple membership random effects models (CCMMrem) to model rater bias and estimate teacher effectiveness. Effect estimates are compared using CTT versus item response theory (IRT) scaling methods and three models (i.e., conventional multilevel…

In traditional studies on value-added indicators of educational effectiveness, students are usually treated as belonging to those schools where they took their final examination. However, in practice, students sometimes attend multiple schools and therefore it is questionable whether this assumption of belonging to the last school they attended…

Eliminating inequity in public education is a central goal of the No Child left Behind (NCLB) act. Controlling for 3rd-grade performance, the impact of English language learner (ELL) status, socioeconomic status (SES), and special education (SPED) status on a cohort's reading performance was investigated from elementary to high school through a…