Toward an Interactive Reading Experience: Deep Learning Insights and Visual Narratives of Engagement and Emotion
Jayasankar Santhosh, Akshay Palimar Pai, Shoya Ishimaru
Abstract
Engagement and emotion are critical components that significantly influence a reader’s experience during a reading task. The level of engagement reflects the extent to which a reader is immersed and attentive to the content, while emotion represents the affective responses evoked by the text. The present study aims to detect the engagement and emotion levels during a reading task by leveraging the power of state-of-the-art deep learning models and investigating the correlations between the engagement levels and emotions. An experiment was conducted involving 18 university students reading 14 documents followed by a questionnaire to rate their levels of engagement, valence, and arousal after reading each document. A Tobii 4C eye-tracker with a pro license along with an Empatica E4 wristband were utilized to record the data from the participants. A range of deep learning models were utilized for computing the engagement, valence, and arousal values, employing both user-independent and user-dependent methods. Notably, the Transformer model exhibited better performance in terms of accuracy and F1-score in the context of user-independent approaches, while the Resnet model excelled when adopting the user-dependent approach. The recorded data was also utilized to establish the relationship between engagement levels and emotional states concerning the specific document being read. The prediction results were utilized to develop an interactive and user-friendly application designed to provide a visual representation of a reader’s engagement level and emotional state while performing a reading task. The dashboard features an engagement gauge that displays the reader’s level of engagement based on predicted class probabilities, and an emotion emoji serving as a visual cue that illustrates the predicted emotional state of the reader.