In-sensor compressing via programmable optoelectronic sensors based on van der Waals heterostructures for intelligent machine vision
Haoxin Huang, Shuhui Shi, Jiajia Zha, Yunpeng Xia, H. Wang, Peng Yang, Long Zheng, Songcen Xu, Wei Wang, Yi Ren, Yongji Wang, Ye Chen, Hau Ping Chan, Johnny C. Ho, Yang Chai, Zhongrui Wang, Chaoliang Tan
Abstract
Efficiently capturing multidimensional signals containing spectral and temporal information is crucial for intelligent machine vision. Although in-sensor computing shows promise for efficient visual processing by reducing data transfer, its capability to compress temporal/spectral data is rarely reported. Here we demonstrate a programmable two-dimensional (2D) heterostructure-based optoelectronic sensor integrating sensing, memory, and computation for in-sensor data compression. Our 2D sensor captured and memorized/encoded optical signals, leading to in-device snapshot compression of dynamic videos and three-dimensional spectral data with a compression ratio of 8:1. The reconstruction quality, indicated by a peak signal-to-noise ratio value of 15.81 dB, is comparable to the 16.21 dB achieved through software. Meanwhile, the compressed action videos (in the form of 2D images) preserve all semantic information and can be accurately classified using in-sensor convolution without decompression, achieving accuracy on par with uncompressed videos (93.18% vs 83.43%). Our 2D optoelectronic sensors promote the development of efficient intelligent vision systems at the edge. In-sensor computing offers a promising solution for image processing with reduced data transfer. Here, the authors report programmable and multifunctional van der Waals optoelectronic sensors, showing their application for snapshot compression and recognition of dynamic videos and 3D spectral data.