Transmodal Analysis

Abstract

Learning is a multimodal process, and learning analytics (LA) researchers can readily access rich learning process data from multiple modalities, including audio-video recordings or transcripts of in-person interactions; logfiles and messages from online activities; and biometric measurements such as eye-tracking, movement, and galvanic skin response. While many techniques are used in LA to model different types of learning process data—most of which are state-dependent (or state-space) approaches that model a learning process at any given time as a function of the preceding events—constructing multimodal models has so far relied on fusion of different data streams, which converts multimodal data into a unimodal format. This creates a number of problems for multimodal modelling, the most important of which is that it treats different data modalities as equivalent. That is, existing state-dependent models of fused data cannot easily account for (a) events that may have different impacts on future events based on what those future events are and the context in which they are occurring; (b) how events may influence some groups of learners differently; and (c) which events are visible (and thus potentially impactful) to which students. In this paper, we propose transmodal analysis (TMA), a mathematical and computational framework designed to address these challenges. TMA is not a data analysis method but rather an approach to modelling that can augment existing state-dependent models of learning processes to account for multimodal data without data fusion. We present a conceptual and methodological description of TMA, and we include an appendix with a detailed worked example as a proof of concept. While this approach is in the early stages of development, it has the potential to significantly improve the ease, efficiency, and fairness of multimodal analyses of learning processes.

Topics & Concepts