Dynamic Item Response Models extend the standard Item Response Theory (IRT) to capture temporal dynamics in learner ability. While these models have the potential to allow instructional systems to actively monitor the evolution of learner proficiency in real time, existing dynamic item response models rely on expensive inference algorithms that…

Surveys with rating scales are often used in higher education research to measure student learning and development, yet testing and reporting on the longitudinal psychometric properties of these instruments is rare. Rasch techniques allow scholars to map item difficulty and individual aptitude on the same linear, continuous scale to compare…

In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this article, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric…

This article presents the results of evaluating a dataset collected with the Community of inquiry (CoI) survey (Arbaugh, The International Review of Research in Open and Distributed Learning 9:1-21, 2008) using Rasch psychometric techniques to evaluate instrument functioning. Data were collected over a two-year period yielding a sample of 704…

Empirical growth benchmarks, as introduced by Hill, Bloom, Black, and Lipsey (2008), are a well-known way to contextualize effect sizes in education research. Past work on these benchmarks, both positive and negative, has largely avoided confronting the role of vertical scales, yet technical issues with vertical scales trouble the use of such…

The R software provides packages and functions that provide data analysis in classical true score, generalizability theory, item response theory, and Rasch measurement theories. A brief list of notable articles in each measurement theory and the first measurement journals is followed by a list of R psychometric software packages. Each psychometric…

In this digital ITEMS module, Dr. Roy Levy describes Bayesian approaches to psychometric modeling. He discusses how Bayesian inference is a mechanism for reasoning in a probability-modeling framework and is well-suited to core problems in educational measurement: reasoning from student performances on an assessment to make inferences about their…

Item Response Theory (IRT) has proven useful in physics education research to examine the validity of concept tests (e.g., Refs. 1-3) and online homework (e.g. Refs. 4-6), yet as a tool for the improvement of physics instruction (particularly exams), it is oftentimes perceived as (a) mysterious, (b) unjustified, and (c) impractical. This article…

This methodological paper examines current conceptions of reliability in chemistry education research (CER) and provides recommendations for moving beyond the current reliance on reporting coefficient alpha (a) as reliability evidence without regard to its appropriateness for the research context. To help foster a better understanding of…

The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal…

The design and psychometric methodology of the National Assessment of Educational Progress (NAEP) is constantly evolving to meet the changing interests and demands stemming from a rapidly shifting educational landscape. NAEP has been built on strong research foundations that include conducting extensive evaluations and comparisons before new…

International large-scale assessments (ILSAs) transitioned from paper-based assessments to computer-based assessments (CBAs) facilitating the use of new item types and more effective data collection tools. This allows implementation of more complex test designs and to collect process and response time (RT) data. These new data types can be used to…

As perfectly summarised by Ida Lawrence, "Testing is growing by leaps and bounds across the world. There is a realization that a nation's well-being depends crucially on the educational achievement of its population. Valid tests are an essential tool to evaluate a nation's educational standing and to implement efficacious educational reforms.…

Examinees' thinking processes have become an increasingly important concern in testing. The responses processes aspect is a major component of validity, and contemporary tests increasingly involve specifications about the cognitive complexity of examinees' response processes. Yet, empirical research findings on examinees' cognitive processes are…

Drawing valid inferences from modern measurement models is contingent upon a good fit of the data to the model. Violations of model-data fit have numerous consequences, limiting the usefulness and applicability of the model. As Bayesian estimation is becoming more common, understanding the Bayesian approaches for evaluating model-data fit models…