A 0.8 V Intelligent Vision Sensor With Tiny Convolutional Neural Network and Programmable Weights Using Mixed-Mode Processing-in-Sensor Technique for Image Classification
Tzu-Hsiang Hsu, Guan-Cheng Chen, Yiren Chen, Ren-Shuo Liu, Chung‐Chuan Lo, Kea‐Tiong Tang, Meng‐Fan Chang, Chih-Cheng Hsieh
Abstract
This article presents an intelligent vision sensor (IVS) with embedded tiny convolutional neural network (CNN) model and programmable processing-in-sensor (PIS) circuit for real-time inference applications of low-power edge devices. The proposed imager realizes the full computing functions of a customized three-layers tiny network, which includes a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3 \times 3$ </tex-math></inline-formula> convolution layer (stride <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$=$ </tex-math></inline-formula> 3) with activation function of rectified linear unit (ReLU), a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2 \times 2$ </tex-math></inline-formula> maximum pooling (MP) layer (stride <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$=$ </tex-math></inline-formula> 2), and a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1 \times 1$ </tex-math></inline-formula> fully connected (FC) layer for inference. A 0.8 V <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$128 \times 128$ </tex-math></inline-formula> IVS prototype was fabricated and verified in TSMC 0.18 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{m}$ </tex-math></inline-formula> standard CMOS technology. In normal image mode, it consumed 76.4 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{W}$ </tex-math></inline-formula> with full-resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$126 \times 126$ </tex-math></inline-formula> active resolution) image output at 125 f/s. In CNN mode, it consumed 134.5 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{W}$ </tex-math></inline-formula> at 250 f/s and an achieved iFoMs of 33.8 pJ/pixel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\cdot $ </tex-math></inline-formula> frame. Using the proposed mixed-mode PIS circuits, the prototype is configured to demonstrate a “human face or not detection” task with an achieved accuracy of 93.6%.