Affective computing (AC) has been regarded as a relevant approach to identifying online learners' mental states and predicting their learning performance. Previous research mainly used one single-source data set, typically learners' facial expression, to compute learners' affection. However, a single facial expression may represent different…

We explore multi-modal machine learning-based approaches (facial expression recognition, auditory emotion recognition, and text sentiment analysis) to identify "negative moments" of teacher-student interaction during classroom teaching. Our analyses on a large (957 videos, each 20min) dataset of classroom observations suggest that: (1)…

Using multimodal data fusion techniques, we built and tested prediction models to track middle-school student distress states during educational gameplay. We collected and analyzed 1,145 data instances, sampled from a total of 31 middle-school students' audio- and video-recorded gameplay sessions. We conducted data wrangling with student gameplay…

Background: Video lectures which include the teachers' presence have become increasingly common. As teacher enthusiasm is a nonverbal cue in video lectures, more and more studies are focusing on this topic. However, little research has been carried out on the interactions between teacher enthusiasm and prior knowledge when learning from video…

In this study, 118 native speakers of Japanese watched 48 separate video clips in which a teacher provided recasts on phonological or lexical errors to students in Portuguese, a language with which the participants were unfamiliar. In the video clips, six recast characteristics were manipulated: length, segmentation (segmented/whole), prosodic…

This study examined image-text relations in English-as-a-foreign-language textbooks focusing on moving-image animations that accompany verbal dialogues. It also explores how animations support verbal dialogues in the textbooks. Data sources include three major versions of junior high school textbooks locally-produced in Taiwan, each consisting of…

Language classroom interactions can be characterized as multimodal, since teachers may resort to a variety of resources provided by their body or by their immediate space in order to convey meaning, manage activities and assess pupils' performances. Furthermore, teachers' multimodal practices constitute an essential component for the…

This study addresses the need for innovative research approaches in science education for understanding better the inter-relationships between emotion and cognition in learning, using a sociological perspective. Our perspective draws upon the concept of emotional energy that is described as an outcome from successful social interactions during…

The aim of this study is to better understand how teachers use various embodied semiotic modes -- speech, gestures, gaze, and proxemics -- when they present scientific content to the entire class. To analyse the multimodal practices of these teachers, we integrated the social semiotic theory of multimodality and the joint action theory in…

In this article we examine a teacher's and multilingual students' use of multiple resources and their potential for students' meaning-making of sound and sound transmission. Students were 14-15 years old, Swedish grade 8, speaking Swedish as a second language. We examine how different strategies and multiple resources interact in creating thematic…

This article presents an analysis of a digital video created by a student (age 13) in a classroom setting. Since sign functioning is a key focus in theories of meaning making as it occurs through language and through other modes, my analysis focuses on the relations between signifiers as they are inscribed in her video. This analysis explores new…

Increasing research on teacher professional development (TPD) has found teachers' self-reflection to be key for improving teaching effectiveness. Although video methodology, as often used in TPD, provides crucial insight concerning situated learning, teachers are often reticent to participate in TPD protocols due to discomfort over being…

Recent educational research emphasises the importance of analysing talk and gestures to come to an understanding about students' conceptual learning. Gestures are perceived as complex hand movements being equivalent to other language modes. They can convey experienceable as well as abstract concepts. As well as technical language, gestures…

In this article, I examine the relation between literacy events and literacy practices in classroom interaction and add to ongoing discussions in the field of NLS about the transcontextual nature of literacy and how local literacy events are linked to broader literacy practices. It specifically focuses on how the link between literacy events and…

Mathematical problem solving typically involves manipulating visual symbols (e.g., equations), and prior research suggests that those symbols serve as diagrammatic representations (e.g., Landy & Goldstone, 2010). The present work examines the ways that instructional design of student engagement with these diagrammatic representations may…