Inference speed and quantisation of neural networks with TensorFlow Lite for Microcontrollers framework
Kristian Đokić, Marko Martinović, Dubravka Mandušić
Abstract
In the last few years, microcontrollers became more and more powerful, and many authors have started to use them for different machine learning projects. One of the most popular frameworks for machine learning is TensorFlow, and their authors began to develop this framework for microcontrollers. The goal of this paper is to analyses the full connected neural networks inference speed depending on the number of neurons of one popular microcontroller (Arduino Nano 33 BLE Sense) with simple neural networks implementation, as well as the impact of neural network weights quantisation. We expected a reduction in the size of the model with the selected quantization by four times, which was achieved, but with a large number of neurons in the neural network. TensorFlow Lite for Microcontrollers is used with the Arduino Integrated Development Environment. Neural networks with two hidden layers are used with a different number of neurons.