An Analysis of Double <i>Q</i>-Learning-Based Energy Management Strategies for TEG-Powered IoT Devices
Michal Prauzek, Jaromír Konecny, Tereza Paterova
Abstract
The study presents a self-learning controller for managing the energy in an Internet of Things (IoT) device powered by energy harvested from a thermoelectric generator (TEG). The device’s controller is based on a double <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>-learning (DQL) method; the hardware incorporates a TEG energy harvesting subsystem with a dc/dc converter, a load module with a microcontroller, and a LoRaWAN communications interface. The model is controlled according to adaptive measurements and transmission periods. The controller’s reward policy evaluates the level of charge available to the device. The controller applies and evaluates various learning parameters and reduces the learning rate over time. Using four years of historical soil temperature data in an experimental simulation of several controller configurations, the DQL controller demonstrated correct operation, a low learning rate, and high cumulative rewards. The best energy management controller operated with a completed cycle and missed cycle ratio of 98.5%. The novelty of the presented approach is discussed in relation to state-of-the-art methods in adaptive ability, learning processes, and practical applications of the device.