Parallel analysis has been considered one of the most accurate methods for determining the number of factors in factor analysis. One major advantage of parallel analysis over traditional factor retention methods (e.g., Kaiser's rule) is that it addresses the sampling variability of eigenvalues obtained from the identity matrix, representing the…

We review common situations in Bayesian latent variable models where the prior distribution that a researcher specifies differs from the prior distribution used during estimation. These situations can arise from the positive definite requirement on correlation matrices, from sign indeterminacy of factor loadings, and from order constraints on…

We explored three approaches to resolving or re-scoring constructed-response items in mixed-format assessments: rater agreement, person fit, and targeted double scoring (TDS). We used a simulation study to consider how the three approaches impact the psychometric properties of student achievement estimates, with an emphasis on person fit. We found…

The fit of an item response model is typically conceptualized as whether a given model could have generated the data. We advocate for an alternative view of fit, "predictive fit", based on the model's ability to predict new data. We derive two predictive fit metrics for item response models that assess how well an estimated item response…

Methods to generate random correlation matrices have been proposed in the literature, but very few instances exist where these correlation matrices are structured or where the statistical properties of the algorithms are known. By relying on the tetrad relation discovered by Spearman and the properties of the beta distribution, an algorithm is…

Assessing the measurement model (MM) of self-report scales is crucial to obtain valid measurements of individuals' latent psychological constructs. This entails evaluating the number of measured constructs and determining which construct is measured by which item. Exploratory factor analysis (EFA) is the most-used method to evaluate these…

Purpose: This study evaluated interrater reliability (IRR) and test-retest stability (TRTS) of seven linguistic measures (percent correct information units, relevance, subject-verb-[object], complete utterance, grammaticality, referential cohesion, global coherence), and communicative success in unstructured conversation and in a story narrative…

Supervisors evaluated counselors-in-training at multiple points during their practicum experience using the Counseling Competencies Scale (CCS; N = 1,070). The CCS evaluations were randomly split to conduct exploratory factor analysis and confirmatory factor analysis, resulting in a 2-factor model (61.5% of the variance explained).

Research on effective teaching has been conducted in a variety of settings for more than 40 years. This study offers direction for future effective teaching research in secondary agricultural education and has implications for career and technical education. Specifically, 142 items consisting of characteristics, behaviors, and/or techniques…

This study investigates the relationships among factor correlations, inter-item correlations, and the reliability estimates of subscores, providing a guideline with respect to psychometric properties of useful subscores. In addition, it compares subscore estimation methods with respect to reliability and distinctness. The subscore estimation…

Popular teamwork assessments have been strongly criticized on the grounds of poor psychometric properties and their disconnect with conceptual models of teamwork. These issues raise concerns with respect to our ability to evaluate efforts devoted to advancing teamwork in academia. We report the development of a teamwork assessment that builds on…

Background: As the maker movement is increasingly adopted into K-12 schools, students are developing new competences in exploration and fabrication technologies. This study assesses learning with these technologies in K-12 makerspaces and FabLabs. Purpose: Our study describes the iterative process of developing an assessment instrument for this…

Triangulation -- a combined use of different assessment methods or sources to evaluate psychological constructs -- is still a rarely used assessment approach in spite of its potential in overcoming inherent constraints of individual assessment methods. This paper uses field test data from a new OECD Study on Social and Emotional Skills to examine…

Certain diversity among team members is beneficial to the growth of an organization. Multiple measures have been proposed to quantify diversity, although little is known about their psychometric properties. This article proposes several methods to evaluate the unidimensionality and reliability of three measures of diversity. To approximate the…

When subscores on a test are reported to the test taker, the appropriateness of reporting them depends on whether they provide useful information above what is provided by the total score. Subscores that fail to do so lack adequate psychometric quality and should not be reported. There are several methods for examining the quality of subscores,…