Automatic Recognition and Augmentation of Attended Objects in Real-time using Eye Tracking and a Head-mounted Display

Michael Barz, Sebastian Kapp, Jochen Kühn, Daniel Sonntag

2021ACM Symposium on Eye Tracking Research and Applications23 citationsDOI

Abstract

Scanning and processing visual stimuli in a scene is essential for the human brain to make situation-aware decisions. Adding the ability to observe the scanning behavior and scene processing to intelligent mobile user interfaces can facilitate a new class of cognition-aware user interfaces. As a first step in this direction, we implement an augmented reality (AR) system that classifies objects at the user’s point of regard, detects visual attention to them, and augments the real objects with virtual labels that stick to the objects in real-time. We use a head-mounted AR device (Microsoft HoloLens 2) with integrated eye tracking capabilities and a front-facing camera for implementing our prototype.

Topics & Concepts

Computer scienceAugmented realityComputer visionEye trackingArtificial intelligencePoint (geometry)Optical head-mounted displayHuman–computer interactionVirtual realityMobile deviceTracking (education)Computer graphics (images)PsychologyGeometryPedagogyMathematicsOperating systemGaze Tracking and Assistive TechnologyAugmented Reality ApplicationsVisual Attention and Saliency Detection