In this paper we will explore an application, made by us, that can generate physics grid tests using artificial intelligence. The application analyzes the response from the large language model in the required format recognized by the application and writes it to a file that will be accepted by the test builder application. The application creates…

Although AI transformer models have demonstrated notable capability in automated scoring, it is difficult to examine how and why these models fall short in scoring some responses. This study investigated how transformer models' language processing and quantification processes can be leveraged to enhance the accuracy of automated scoring. Automated…

The Homeostasis Concept Inventory (HCI) is a validated instrument for measuring students' knowledge of homeostasis. It is comprised of 20 multiple-choice questions covering key components of the previously validated Homeostasis Conceptual Framework (HCF). In this paper, we present the first multi-institutional study of the impact of physiology…

In this study, the performance of artificial intelligence chatbots--OpenAI's ChatGPT, Google Gemini, and Microsoft's Copilot--was evaluated and compared based on their responses to questions from the Turkish Higher Education Entrance Physics Examination over the past three years. Analysis of the chatbots' responses to TYT Physics questions showed…

Limited research has explored the use of visual displays (ViDis) in science tests, making it challenging to know how these tests align with classroom instruction and what skills students need to be successful on these tests. Therefore, the current study aims to describe the use of ViDis in upper elementary grade standardized science tests. We…

Representations are an integral component of organic chemistry education and practice. Learning how to visualize and reason effectively with a variety of representations is essential for learners to succeed. However, developing the necessary representational competence skills (e.g., the ability to interpret, translate between, and use…

We investigate the multilingual and multimodal performance of a large language model-based artificial intelligence (AI) system, GPT-4o, using a diverse set of physics concept inventories spanning multiple languages and subject categories. The inventories, sourced from the PhysPort website, cover classical physics topics such as mechanics,…

Knowledge of gross anatomy and embryology continues to be important for medical student training. However, the relevance of anatomical knowledge during the preclinical years remains underexplored considering recent revisions to the scoring of the United States Medical Licensing Examination (USMLE). Guided by the framework of cognitive…

Natural category learning is important in science education. One strategy that has been empirically supported for enhancing category learning is testing, which facilitates not only the learning of previously studied information (backward testing effect) but also the learning of newly studied information (forward testing effect). However, in…

The artificial intelligence chatbot ChatGPT was used to answer questions from final exams administered in two general chemistry courses, including questions with closed-response format and with open-response format. For closed-response questions, ChatGPT was very capable at identifying the concept even when the question included a great deal of…

This study aimed to develop a comprehensive diagnostic tool for assessing upper-secondary school students' understanding of isomers, expanding upon existing two- and three-tier conceptual diagnostic methods. By incorporating 'Confidence Rating Factor' tiers within the answer and reason sections, a four-tier test was designed and developed. This…

Laboratory learning has been the focus of significant creativity for educators. Engaging students in "real world" contexts or using their knowledge in laboratory exercises in a meaningful way requires that students are engaged in the practice of science. The role of decision-making around this practice of science, say for the analysis of…

Preparing students for science, technology, and engineering careers is an urgent state policy challenge. We examine the design and roll-out of a science testing requirement for high-school graduation in Massachusetts. While science test performance has improved over time for all demographic subgroups, we observe rising inequality in failure rates…

The Force Concept Inventory (FCI) is one of the research-based assessments established by the physics education research community to measure students' understanding of Newtonian mechanics. Former works have often recorded the notion of gendered mean FCI scores favoring male students notably in the North American (NA) based studies. Nevertheless,…

This article provides a process to carefully evaluate the suitability of a content domain for which diagnostic classification models (DCMs) could be applicable and then optimized steps for constructing a test blueprint for applying DCMs and a real-life example illustrating this process. The content domains were carefully evaluated using a set of…