Purpose: Many studies using machine learning (ML) in speech, language, and hearing sciences rely upon cross-validations with single data splitting. This study's first purpose is to provide quantitative evidence that would incentivize researchers to instead use the more robust data splitting method of nested k-fold cross-validation. The second…

Psychology research often focuses on interactions, and this has deep implications for inference from non-representative samples. For the goal of estimating average treatment effects, we propose to fit a model allowing treatment to interact with background variables and then average over the distribution of these variables in the population. This…

Phonotactics--constraints on the position and combination of speech sounds within syllables--are subject to statistical differences that gradiently affect speaker and listener behavior (e.g., Vitevitch & Luce, 1999). What statistical properties drive the acquisition of such constraints? Because they are naturally highly correlated, previous…

Recommended learning goals for students in introductory statistics courses include the ability to recognize and explain the key role of randomness in designing studies and in drawing conclusions from those studies involving generalizations to a population or causal claims (GAISE College Report ASA Revision Committee, 2016). The purpose of this…

Second language acquisition researchers often face particular challenges when attempting to generalize study findings to the wider learner population. For example, language learners constitute a heterogeneous group, and it is not always clear how a study's findings may generalize to other individuals who may differ in terms of language background…

Online education continues to become an increasingly prominent part of higher education, but many students struggle in distance courses. For this reason, there has been considerable interest in predicting which students will succeed in online courses and which will receive poor grades or drop out prior to completion. Effective intervention depends…

Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the…

How parents interact with their children impacts many crucial facets of children's lives. Over the last 4 decades, researchers have identified four different parenting styles: authoritative, authoritarian, permissive, and disengaged. Hundreds of studies conducted all over the world, have identified correlations between parenting style and many…

The report is the first systematic evaluation of the sentence equivalence item type introduced by the "GRE"® revised General Test. We adopt a validity framework to guide our investigation based on Kane's approach to validation whereby a hierarchy of inferences that should be documented to support score meaning and interpretation is…

In this study, we revised the "Metacognitive Awareness of Reading Strategies Inventory" (MARSI), a self-report instrument designed to assess students' awareness of reading strategies when reading school-related materials. We collected evidence of structural, generalizability, and external aspects of validity for the revised inventory…

This research provides new insights into the measurement of students' authorial identity and its potential for minimising the incidence of unintentional plagiarism by providing evidence about the psychometric properties of the Student Authorship Questionnaire (SAQ). Exploratory and confirmatory factor analyses (EFA and CFA) are employed to…

Since heterogeneity between reliability coefficients is usually found in reliability generalization studies, moderator analyses constitute a crucial step for that meta-analytic approach. In this study, different procedures for conducting mixed-effects meta-regression analyses were compared. Specifically, four transformation methods for the…

This work has a three-fold purpose. The first is to set out the rationale for an alternative approach for assessing the generalizability of research findings. (It is related to but not equivalent to the CE method.) The second is to formalize this approach through a quantitative model. The third is to apply the approach to the results of the…

The past two decades have shown an unexpected effectiveness of "Web-scale" data in natural language processing. Even the simplest models, when paired with unprecedented amounts of unstructured and unlabeled Web data, have been shown to outperform sophisticated ones. It has been argued that the effectiveness of Web-scale data has…

Comparing regression coefficients between models when one model is nested within another is of great practical interest when two explanations of a given phenomenon are specified as linear models. The statistical problem is whether the coefficients associated with a given set of covariates change significantly when other covariates are added into…