Meta-analyses of educational research findings frequently involve statistically dependent effect size estimates. Meta-analysts have often addressed dependence issues using ad hoc approaches that involve modifying the data to conform to the assumptions of models for independent effect size estimates, such as by aggregating estimates to obtain one…

In recent years, educators, administrators, policymakers, and measurement experts have called for assessments that support educators in making better instructional decisions. One promising approach to measurement to support instructional decision-making is diagnostic classification models (DCMs). DCMs are flexible psychometric models that…

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two…

With the popularity of online one-to-one tutoring, there are emerging concerns about the quality and effectiveness of this kind of tutoring. Although there are some evaluation methods available, they are heavily relied on manual coding by experts, which is too costly. Therefore, using machine learning to predict instruction quality automatically…

The Campbellian validity typology has been used as a foundation for outcome evaluation and for developing evidence-based interventions for decades. As such, randomized control trials were preferred for outcome evaluation. However, some evaluators disagree with the validity typology's argument that randomized controlled trials as the best design…

Policymakers have expressed increased interest in program-level higher education accountability measures as a supplement to, or in place of, institution-level metrics. But it is unclear what these measures should look like. In this report, we assess the ways program-level data could be developed to facilitate federal accountability. Evidence shows…

Classification of higher education institutions (HEIs) of a country allows viewing higher education as a differentiated system which respects the diversity of purposes and aspirations of different HEIs. Classification is fundamentally different from ranking, which aims to rank universities in order with higher ranked HEIs being "better"…

Advanced modeling of item response data through the item response theory (IRT) or item factor analysis frameworks is becoming increasingly popular. In the social and behavioral sciences, the underlying structure of tests/assessments is often multidimensional (i.e., more than 1 latent variable/construct is represented in the items). This review…

In 2013, the Council of Chief State School Officers (CCSSO), working collaboratively with state education agencies, released a set of criteria for states to use to evaluate and procure high-quality assessments. The mathematics section of the document included five content-specific criteria to evaluate alignment of assessments to college- and…

In this paper, a quantitative system of undergraduates' creative ability is proposed through analyzing characteristics of Amabile creative theory, and the objectivity and feasibility of CAT, TTCT and AMS in creative ability quantification. The academic test scores, TTCT scores and AMS scores are used as the quantitative index of professional…

This commentary will take an historical perspective on the Kaufman Test of Educational Achievement (KTEA) error analysis, discussing where it started, where it is today, and where it may be headed in the future. In addition, the commentary will compare and contrast the KTEA error analysis procedures that are rooted in psychometric methodology and…

This paper argues that the dominant framework for conceptualizing validation evidence and analysis--the "five sources" framework from the 1999 "Standards"--is seriously limited. Its limitation raises a significant barrier to understanding the nature of comprehensive validation, and this presents a significant threat to…

The requirement to assure the teaching of critical thinking put the school in front of its own weaknesses. A profound criticism highlights limitations, hindrances and obstacles that are difficult to pass without the personal efforts of the teachers. Following criticism, one can identify a set of requirements that would allow for improvement and…

The significant amount of information available on the web has led to difficulties for the learner to find useful information and relevant resources to carry out their training. The recommender systems have achieved significant success in the area of e-commerce, they still have difficulties in formulating relevant recommendations on e-learning…

Assessing global interrater agreement is difficult as most published indices are affected by the presence of mixtures of agreements and disagreements. A previously proposed method was shown to be specifically sensitive to global agreement, excluding mixtures, but also negatively biased. Here, we propose two alternatives in an attempt to find what…