Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning
Yawen Wu, Zhepeng Wang, Yiyu Shi, Jingtong Hu
Abstract
This work aims to enable on-device training of convolutional neural networks (CNNs) by reducing the computation cost at training time. CNN models are usually trained on high-performance computers and only the trained models are deployed to edge devices. But the statically trained model cannot adapt dynamically in a real environment and may result in low accuracy for new inputs. On-device training by learning from the real-world data after deployment can greatly improve accuracy. However, the high computation cost makes training prohibitive for resource-constrained devices. To tackle this problem, we explore the computational redundancies in training and reduce the computation cost by two complementary approaches: 1) self-supervised early instance filtering on data level and 2) error map pruning (EMP) on the algorithm level. The early instance filter selects important instances from the input stream to train the network and drops trivial ones. The EMP further prunes out insignificant computations when training with the selected instances. Extensive experiments show that the computation and energy cost is substantially reduced without any or with marginal accuracy loss. For example, when training ResNet-110 on CIFAR-10, we achieve 67.8% computation saving while preserving full accuracy and 75.1% computation saving with a marginal accuracy loss of 1.3%. When training LeNet on MNIST, we save 79% computation while boosting accuracy by 0.2%. Besides, practical energy saving is measured on edge platforms. We achieve 67.6% energy saving when training ResNet-110 on mobile GPU and 74.1% energy saving when training LeNet on MCU without any accuracy loss.