Haptic-Captioning: Using Audio-Haptic Interfaces to Enhance Speaker Indication in Real-Time Captions for Deaf and Hard-of-Hearing Viewers
Yiwen Wang, Ziming Li, Pratheep Kumar Chelladurai, Wendy Dannels, Tae Oh, Roshan Lalintha Peiris
Abstract
Captions make the audio content of videos accessible and understandable for deaf or hard-of-hearing people (DHH). However, in real-time captioning scenarios, captions alone can be challenging for DHH users to identify the active speaker in a real time in multiple-speaker scenarios. To enhance the accessibility of real-time captioning, we propose Haptic-Captioning which provides real-time vibration feedback on the wrist by directly translating the sound of content into vibrations. We conducted three experiments to examine: (1) the haptic perception (Preliminary Study), (2) the feasibility of the haptic modality along with real-time and non-real-time visual captioning methods (Study 1), and (3) the user experience of using the Haptic-Captioning system in different media contexts (Study 2). Our results highlight that the Haptic-Captioning complements visual captions by improving caption readability, maintaining media engagement, enhancing understanding of emotions, and assisting speaker indication in real-time captioning scenarios. Furthermore, we discuss design implications for the future development of Haptic-Captioning.