The article reports three simulation studies conducted to find out whether the effect of a time limit for testing impairs model fit in investigations of structural validity, whether the representation of the assumed source of the effect prevents impairment of model fit and whether it is possible to identify and discriminate this method effect from…

This inquiry is focused on three indicators of the precision of measurement--conditional on fixed values of ?, the latent variable of item response theory (IRT). The indicators that are compared are (1) The traditional, conditional standard errors, s(eX|?) = CSEM; (2) the IRT-based conditional standard errors, s[subscript irt](eX|?)=C[subscript…

One of the important theories in education and psychology is Generalizability (G) Theory and various properties distinguish it from the other measurement theories. To better understand methodological trends of G theory, a thematic content analysis was conducted. This study analyzes the studies using generalizability theory in the field of…

Applying single-level statistical models to multilevel data typically produces underestimated standard errors, which may result in misleading conclusions. This study examined the impact of ignoring multilevel data structure on the estimation of item parameters and their standard errors of the Rasch, two-, and three-parameter logistic models in…

When estimating an average treatment effect with observational data, it's possible to get an unbiased estimate of the causal effect if all confounding variables are observed and reliably measured. In education, confounding variables are often latent constructs. Covariate selection methods used in causal inference applications assume that all…

When latent variables are used as outcomes in regression analysis, a common approach that is used to solve the ignored measurement error issue is to take a multilevel perspective on item response modeling (IRT). Although recent computational advancement allow efficient and accurate estimation of multilevel IRT models, we argue that a two-stage…

The relations among alternative parameterizations of the binary factor analysis (FA) model and two-parameter logistic (2PL) item response theory (IRT) model have been thoroughly discussed in literature (e.g., Lord & Novick, 1968; Takane & de Leeuw, 1987; McDonald, 1999; Wirth & Edwards, 2007; Kamata & Bauer, 2008). However, the…

Questions on voluntary association memberships have been used extensively in social scientific research for decades. Researchers generally assume that these respondent self-reports are accurate, but their measurement has never been assessed. Respondent characteristics are known to influence the accuracy of other self-report variables such as…

The purpose of this research is to examine measurement invariance of collaborative problem solving skills measured by PISA 2015 Xandar subtest for Singapore, Norway, and Turkey. The research was conducted with 2990 participants' data obtained from Turkey (1032), Norway (923), and Singapore (1035) on PISA 2015 collaborative problem solving study.…

Temporality-driven covariate classification had limited impact on: the specification of directed acyclic graphs (DAGs) by 85 novice analysts (medical undergraduates); or the risk of bias in DAG-informed multivariable models designed to generate causal inference from observational data. Only 71 students (83.5%) managed to complete the…

Background/Context: School principals have long played a central role in managing building operations and supervising school employees. Over the past two decades, however, policy developments--including site-based management, external accountability measures, and teacher evaluation systems--have increased expectations that principals also improve…

The finite population correction (FPC) factor is often used to adjust variance estimators for survey data sampled from a finite population without replacement. As a replicated resampling approach, the jackknife approach is usually implemented without the FPC factor incorporated in its variance estimates. A paradigm is proposed to compare the…

Scholarly debate about student evaluations of teaching (SETs) often focuses on whether SETs are valid, reliable and unbiased. In this article, we assume the most optimistic conditions for SETs that are supported by the empirical literature. Specifically, we assume that SETs are moderately correlated with teaching quality (student learning and…

Appropriate interpretation of assessment data requires an appreciation that tools are subject to measurement error. School psychologists recognize, at least on an intellectual level, that measures are imperfect--that test scores and other quantitative measures (e.g., rating scales, systematic behavioral observations) are best estimates of…

Despite the importance of identifying individuals with reading disabilities, existing operational definitions of reading disability do not result in reliable identification. A large part of the problem arises from measurement error when a cut-point is imposed on a continuous distribution, especially for low base-rate conditions. One way to reduce…