A 25.1-TOPS/W Sparsity-Aware Hybrid CNN-GCN Deep Learning SoC for Mobile Augmented Reality
Wen-Cong Huang, I-Ting Lin, Ying-Sheng Lin, Wen-Ching Chen, Liang-Yi Lin, Nian-Shyang Chang, Chun‐Pin Lin, Chi‐Shi Chen, Chia‐Hsiang Yang
Abstract
Augmented reality (AR) has been applied to various mobile applications. Modern AR algorithms include neural networks, such as convolutional neural networks (CNNs) and graph convolutional networks (GCNs). The high computational complexity of these networks poses challenges for real-time operation on energy-constrained devices. This article presents the first energy-efficient hybrid CNN-GCN system-on-chip (SoC) for mobile AR. A CNN engine exploits the channel-wise structured feature sparsity to eliminate redundant computations and data movements. By utilizing the proposed channel-sparse encoding scheme on a specialized processing element (PE) architecture, up to 8 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> higher throughput and 6.1 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> higher energy efficiency can be achieved. A reconfigurable convolution PE (CPE) array is deployed for efficient CNN inference. A GCN engine is designed to implement skeleton-based action recognition and gesture recognition. Up to 71% of total operations and 39% of memory footprint can be reduced by leveraging the data and graph properties. A RISC-V MCU is integrated for system control and network deployment. The proposed SoC is implemented in a 28-nm CMOS technology with a core area of 8.28 mm <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^2$</tex-math> </inline-formula> . By exploiting various sparsity levels across network layers, the chip achieves an up to 3.277-TOPS peak performance and a 25.1-TOPS/W energy efficiency for sparse CNN inference, 2.0 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> higher energy-efficient than prior arts. It also achieves an up to 72 actions/s recognition throughput, 18 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> faster than the state of the art.