Learning to interpret data in context is an important educational outcome. To assess students' attainment of this outcome, it is necessary to examine the interplay between their contextual and statistical reasoning. We describe a research method designed to do so. The method draws upon Toulmin's (1958, 2003) model of argumentation for the first…

High dropout rate exists universally in massive open online courses (MOOCs) due to the separation of teachers and learners in space and time. Dropout prediction using the machine learning method is an extremely important prerequisite to identify potential at-risk learners to improve learning. It has attracted much attention and there have emerged…

Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim…

To provide personalized support on educational platforms, it is crucial to model the evolution of students' knowledge states. Knowledge tracing is one of the most popular technologies for this purpose, and deep learning-based methods have achieved state-of-the-art performance. Compared to classical models, such as Bayesian knowledge tracing, which…

Evidence-based assessment (EBA) requires that investigators employ scientific theories and research findings to guide decisions about what domains to measure, how and when to measure them, and how to make decisions and interpret results. To implement EBA, investigators need high-quality assessment tools along with evidence-based processes. We…

Several programs of research have sought to assess the replicability of scientific findings in different fields, including economics and psychology. These programs attempt to replicate several findings and use the results to say something about large-scale patterns of replicability in a field. However, little work has been done to understand the…

Education systems are increasingly creating rich, longitudinal data sets with frequent, and even real-time, data updates of many student measures, including daily attendance, homework submissions, and exam scores. These data sets provide an opportunity for district and school staff members to move beyond an indicators-based approach and instead…

Researchers use many different metrics for evaluation of performance of student models. The aim of this paper is to provide an overview of commonly used metrics, to discuss properties, advantages, and disadvantages of different metrics, to summarize current practice in educational data mining, and to provide guidance for evaluation of student…

In light of the emergence of statistical analysis techniques based on data mining in education sciences, and the potential they offer to detect non-trivial information in large databases, this paper presents a procedure used to detect factors linked to academic achievement in large-scale assessments. The study is based on a non-experimental,…

This dissertation comprises three papers that propose, discuss, and illustrate models to make improved inferences about research questions regarding student achievement in education. Addressing the types of questions common in educational research today requires three different "extensions" to traditional educational assessment: (1)…

The purpose of this study was to analyze how school psychologists engaged in problem analysis during problem-solving consultation. Five aspects of the problem analysis process were examined: 1) the types of questions participants asked during problem identification, 2) the types of data participants requested, 3) the frequency of requests for each…

This paper introduces five commonly used approaches to analyzing multiple-choice test data. They are classical test theory, factor analysis, cluster analysis, item response theory, and model analysis. Brief descriptions of the goals and algorithms of these approaches are provided, together with examples illustrating their applications in physics…

In the modern world, higher education is transitioning from enrollment mode to recruitment mode. This shift paved the way for institutional research and policy making from historical data perspective. More and more universities in the U.S. are implementing and using enterprise resource planning (ERP) systems, which collect vast amounts of data.…

The Office of Planning, Research and Evaluation (OPRE), a unit within the Administration for Children and Families (ACF), is responsible for advising the Assistant Secretary for Children and Families on increasing the effectiveness and efficiency of programs to improve the economic and social well-being of children and families. In collaboration…

This paper presents a framework for monitoring implementation which defines a process for implementing new programs into organizations. The process requires a team of monitors who examine the adequacy of implementation plans and look at the effect of plan execution upon the organization, individual, and new program. Immediate feedback is provided…