We compared the influence of open-book extended duration versus closed book time-limited format on reliability and validity of written assessments of pharmacology learning outcomes within our medical and dental courses. Our dental cohort undertake a mid-year test (30xfree-response short answer to a question, SAQ) and end-of-year paper (4xSAQ,…

Applying qualitative comparative analysis (QCA) to large Ns relaxes researchers' case-based knowledge. This is problematic because causality in QCA is inferred from a dialogue between empirical, theoretical, and case-based knowledge. The lack of case-based knowledge may be remedied by various robustness tests. However, being a case-based method,…

Behavior analysts frequently use the Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP) to assess the language and social skills of children with autism in everyday practice and in research. Despite the widespread use of the VB-MAPP, its psychometric characteristics have not been extensively investigated. To provide information…

The internal validity of observational study is often subject to debate. In this study, we define the counterfactuals as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST). We propose the probability of a robust inference for internal validity, that is, the PIV, as a robustness index…

Item analysis is performed by developers as an integral part of the scale development process. Thus, items are excluded from the scale depending on the item analysis prior to the factor analysis. Existing item discrimination indices are calculated based on correlation, yet items with different response patterns are likely to have a similar item…

Problem decomposition into sub-problems or subgoals and recomposition of the solutions to the subgoals into one complete solution is a common strategy to reduce difficulties in structured problem solving. In this study, we use a datadriven graph-mining-based method to decompose historical student solutions of logic-proof problems into Chunks. We…

Testing English writing skills could be multi-dimensional; thus, the study aimed to compare students' writing scores calculated according to Classical Test Theory (CTT) and Multi-Facet Rasch Model (MFRM). The research was carried out in 2019 with 100 university students studying at a foreign language preparatory class and four experienced…

In Comparative Judgement (CJ) exercises, examiners are asked to look at a selection of candidate scripts (with marks removed) and order them in terms of which they believe display the best quality. By including scripts from different examination sessions, the results of these exercises can be used to help with maintaining standards. Results from…

Reading comprehension is one of the most valuable academic skills taught in school. Selecting the appropriate assessment instrument to ensure early identification and intervention is important as there is an amalgam of cognitive abilities and academic skills involved in reading comprehension. The GORT-5 is the most recent edition of a test that…

Despite the increasing demand for diagnostic information, observed subscores have been often reported to lack adequate psychometric qualities such as reliability, distinctiveness, and validity. Therefore, several statistical techniques based on CTT and IRT frameworks have been proposed to improve the quality of subscores. More recently, DCM has…

Learning analytics researchers have been diligently integrating trace data to study Self-Regulated Learning (SRL). Compared to traditionally used survey data, trace data, such as log or clickstream data designed and interpreted to understand a certain SRL construct, are considered to be more effective in capturing dynamic SRL as fine-grained…

Single-response situational judgment tests (SRSJTs) differ from multiple-response SJTs (MRSJTS) in that they present test takers with edited critical incidents and simply ask test takers to read over the action described and evaluate it according to its effectiveness. Research comparing the reliability and validity of SRSJTs and MRSJTs is thus far…

Organization, time management, and planning (OTMP) skills are behavioral manifestations of executive functioning linked to academic outcomes. Interventions to improve OTMP skills have shown favorable outcomes. The Children's Organizational Skills Scale parent and teacher forms (COSS-P, COSS-T) are widely used for assessing OTMP skills, but there…

Organization, time management, and planning (OTMP) skills are behavioral manifestations of executive functioning linked to academic outcomes. Interventions to improve OTMP skills have shown favorable outcomes. The Children's Organizational Skills Scale parent and teacher forms (COSS-P, COSS-T) are widely used for assessing OTMP skills, but there…

Using an established global academic interest scale, academic interest dimensions of undergraduate (n = 326) and graduate (n = 401) students were compared. The four-factor structure, consisting of passion for learning, confidence in the future, career aspiration, and self-expression, held for both student groups. However, the item scores and mean…