Efficient Execution of Deep Neural Networks on Mobile Devices with NPU

Abstract

Many Deep Neural Network (DNN) based applications have been developed and run on mobile devices. Although these advanced DNN models can provide better results, they also suffer from high computational overhead which means long delay and more energy consumption when running on mobile devices. To address these problems, many companies have developed dedicated Neural Processing Units (NPUs) for mobile devices, which can process AI features. Compared to CPU, NPU can run DNN models much faster, but with lower accuracy. To address this issue, we leverage model partition techniques to improve the performance of DNN models on mobile devices with NPU. The challenge is to determine which part of the DNN model should be run on CPU and which part to be run on NPU. Based on the delay and the accuracy requirements of the applications, we study two problems: Max-Accuracy where the goal is to maximize the accuracy under some time constraint, and Min-Time where the goal is to minimize the processing time while ensuring the accuracy is above a certain threshold. To solve these problems, we propose heuristic based algorithms which are simple but only search a small number of layer combinations (i.e., where to run which DNN model layers). To further improve the performance, we propose a Machine Learning based Model Partition (MLMP) algorithm. MLMP searches more layer combinations and considers both accuracy loss and processing time simultaneously. We also address many implementation issues to support model partition techniques on mobile devices with NPU. Experimental results show that MLMP outperforms the heuristic based algorithms and it can significantly improve the accuracy or reduce the processing time based on the application requirements.

Topics & Concepts