A wearable obstacle avoidance device for visually impaired individuals with cross-modal learning
Yun Gao, Dan Wu, Jie Song, Xueyi Zhang, B. W. Hou, Hengfa Liu, Junqi Liao, Liang Zhou
Abstract
It is challenging for wearable obstacle avoidance devices to simultaneously meet practical demands of high reliability, rapid response, long-lasting duration, and usable design. Here we report a wearable obstacle avoidance device, comprising a set of self-developed glasses (weighing ~400 grams, including an ~80 grams battery) and a common smartphone. Specifically, the glasses collect the multi-modal data for comprehensive environmental perception, including video and depth modalities, and implement a depth-aided video compression module. This module not only adaptively compresses video data to reduce transmission delay to the smartphone, but also operates on a customized FPGA board featuring a multi float-point vector unit streaming processing architecture, thereby facilitating responsive and energy-efficient obstacle detection. Additionally, we design a cross-modal obstacle detection module on the smartphone, which ensures reliable detection and provides user-friendly auditory and tactile alerts by utilizing cross-modal learning based on modal correlations. Multiple indoor and outdoor experimental results demonstrate 100% collision avoidance rates, delay of less than 320 ms, and duration of approximately 11 hours. It is challenging for wearable obstacle avoidance devices to high reliability, rapid response, long-lasting duration, and comfort. Here, the authors report a wearable obstacle avoidance device, comprising a set of self-developed glasses and a common smartphone.