Drawing parallels to classical test theory, this article clarifies the difference between rater accuracy and reliability and demonstrates how category marginal frequencies affect rater agreement and Cohen's kappa. Category assignment paradigms are developed: comparing raters to a standard (index) versus comparing two raters to one another…

Numerous studies on employee engagement in the HRD (human resource development) field have contributed to the promotion of employee engagement in organizations, yet issues and challenges remain. The purpose of this study is to investigate how employee engagement has been discussed in four representative HRD journals: "Human Resource…

Assessing global interrater agreement is difficult as most published indices are affected by the presence of mixtures of agreements and disagreements. A previously proposed method was shown to be specifically sensitive to global agreement, excluding mixtures, but also negatively biased. Here, we propose two alternatives in an attempt to find what…

The Dimensional Change Card Sort (DCCS) task is a widely used measure of preschoolers' executive function. We combined data for 3,290 3-year-olds from 37 unique studies reporting 130 experimental conditions. Using raw pass/fail counts, we computed the pass rates and chi-squared value for each against chance (50/50) performance. We grouped data…

Molenaar extended Mokken's original probabilistic-nonparametric scaling models for use with polytomous data. These polytomous extensions of Mokken's original scaling procedure have facilitated the use of Mokken scale analysis as an approach to exploring fundamental measurement properties across a variety of domains in which polytomous ratings are…

Text classification typically performs best with large training sets, but short texts are very common on the World Wide Web. Can we use resampling and data augmentation to construct larger texts using similar terms? Several current methods exist for working with short text that rely on using external data and contexts, or workarounds. Our focus is…

Americans are increasingly concerned about academic quality (Hauptman & Kim, 2009). This concern about academic quality is a product of the relative decline of the American economy, American Program for International Student Assessment (PISA) test scores, American baccalaureate attainment rates, and various other indicators of national…

This study aims to compare the performances of the artificial neural network, decision trees and discriminant analysis methods to classify student achievement. The study uses multilayer perceptron model to form the artificial neural network model, chi-square automatic interaction detection (CHAID) algorithm to apply the decision trees method and…

The main objective of the article is to compare several methods of evaluating multiple-choice options through classical item analysis. The methods subjected to examination include the tabulation of choice distribution, the interpretation of trace lines, the point-biserial correlation, the categorical analysis of trace lines, and the investigation…

Ambiguity in discourse is pervasive, yet mechanisms of production and processing suggest that it tends to be compensated in context. The present study sets out to analyze the combination of discourse markers (such as "but" or "moreover") with other discourse signals (such as semantic relations or punctuation marks) across three…

Translation teachers have long experimented with various methods to help students improve their translation competence. This study approaches the issue by developing an assessment framework based on error analysis and a translation grading system, with the aim of identifying the most common and frequent errors committed by students in their…

The incorporation of critical pedagogical (CP) perspectives into ELT classrooms has been widely discussed in ELT literature, but how English language teachers in Indonesian schools integrate them in their lessons has not been sufficiently documented. This study aims to investigate to what extent CP perspectives are integrated within the learning…

A driving factor in designing interactive museum exhibits to support simultaneous users is that visitors learn from one another via observation and conversation. Researchers typically analyze such collaborative interactions among museumgoers through manual coding of live- or video-recorded exhibit use. We sought to determine how log data from an…

Principal component analysis (PCA) is one of the most important and powerful methods in chemometrics as well as in a wealth of other areas. Running a PCA results in two main elements, the score plot and the loading plot; the score plot provides the location of the samples, and the loading plot indicates correlations among variables, the trends in…

Growing interest in online learning has instructors looking for new ways to engage learners in asynchronous discussions. Building on experiences with hashtag use to connect on social media platforms, the purpose of this study is to investigate the contexts of hashtag use and its associated impact on learner engagement in asynchronous online…