AoI minimization in status update control with energy harvesting sensors
Mohammad Hatami, Markus Leinonen, Marian Codreanu
Abstract
Information freshness is crucial for time-critical IoT applications, e.g., monitoring and control. We consider an IoT status update system with users, energy harvesting sensors, and a cache-enabled edge node. The users receive time-sensitive information about physical quantities, each measured by a sensor. Users demand for the information from the edge node whose cache stores the most recently received measurements from each sensor. To serve a request, the edge node either commands the sensor to send an update or retrieves the aged measurement from the cache. We aim at finding the best actions of the edge node to minimize the average AoI of the served measurements at the users, termed on-demand AoI. We model this problem as a Markov decision process and develop reinforcement learning (RL) algorithms: model-based value iteration and model-free Q-learning. We also propose a Q-learning method for the realistic case where the edge node is informed about the sensors’ battery levels only via the status updates. The case under transmission limitations is also addressed. Furthermore, properties of an optimal policy are characterized. Simulation results show that an optimal policy is a threshold-based policy and that the proposed RL methods significantly reduce the average cost compared to several baselines.