In recent years, the application of explainability techniques to automated essay scoring and automated short-answer grading (ASAG) models, particularly those based on transformer architectures, has gained significant attention. However, the reliability and consistency of these techniques remain underexplored. This study systematically investigates…

This article builds upon a proiminent definition of construct validity that focuses on variation in attributes causing variation in measurement outcomes. This article synthesizes the defintion and uses Rasch measurement modeling to explicate a modified conceptualization of construct validity for assessments of developmental attributes. If…

Purpose: This study aims to assess the status of sustainable performance, given the significance of indoor air quality related to health and the environment. This research focus on the current status of indoor air quality management in these universities and simplifies its relevance and criticality in safeguarding the well-being of the academic…

Conventional advice discourages controlling for postoutcome variables in regression analysis. By contrast, we show that controlling for commonly available postoutcome (i.e., future) values of the treatment variable can help detect, reduce, and even remove omitted variable bias (unobserved confounding). The premise is that the same unobserved…

We develop a structural after measurement (SAM) method for structural equation models (SEMs) that accommodates missing data. The results show that the proposed SAM missing data estimator outperforms conventional full information (FI) estimators in terms of convergence, bias, and root-mean-square-error in small-to-moderate samples or large samples…

Causal inference regarding a hypothesized mediation mechanism relies on the assumptions that there are no omitted pretreatment confounders (i.e., confounders preceding the treatment) of the treatment-mediator, treatment-outcome, and mediator-outcome relationships, and there are no posttreatment confounders (i.e., confounders affected by the…

The federal Race to the Top competition provided significant impetus for states to adopt value-added models as a part of their teacher evaluation systems. Such models typically link students to their teachers in the spring semester when statewide tests are administered and estimate a teacher's performance based on his or her students' learning…

Attribute-framing bias (AFB) refers to addressees' bias in evaluating positively framed objects (80% success) more favorably than negatively framed ones (20% failure), although they are logically equivalent. The novelty of the current study is in examining conditions in which AFB occurs or does not occur. Typically, AFB is examined for favorable…

The terms "outcome" and "impact" are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a…

Because of the lack of tools available to assess problem-solving skills, teachers often revert to more traditional instructional approaches (e.g. lecture-based, memorization) that fail to prepare learners for the complexity of dynamic work environments. To overcome this challenge, technology solutions are needed that accurately and efficiently…

At present, there is a huge amount of unstructured documentation that is generated by students who study the course project management and evaluation. These unstructured documents cannot be used in direct processing to extract useful information or knowledge. At the same time, make use of them to assess student achievement. In this work, we…

Assessment studies using structural equation modeling and Rasch Model have long been prominent in all areas of educational psychology. Their emphases have covered, but have not been limited to, the improvement of quality in teaching, learning, and research as well as the pursuit of research productivity. However, such methodological approaches…

Culturally responsive evaluation contests that Western-derived evaluation methods represent a universal truth and promotes approaches that reflect the local context. Taking this approach, we examine how the method of contribution analysis may be modified to reflect a different worldview. We reframe contribution analysis using a Confucian lens as…

To assist new scientists in the transition to independent research careers, the National Institutes of Health (NIH) implemented an Early Stage Investigator (ESI) policy beginning with applications submitted in 2009. During the review process, the ESI designation segregates applications submitted by investigators who are within 10 years of…

The present article considers a fundamental question in evaluation research: "By how much do program effects vary across sites?" The article first presents a theoretical model of cross-site impact variation and a related estimation model with a random treatment coefficient and fixed site-specific intercepts. This approach eliminates…