Optimizing AoI in UAV-RIS-Assisted IoT Networks: Off Policy Versus On Policy
Michelle Sherman, Sihua Shao, Xiang Sun, Jun Zheng
Abstract
In urban environments, tall buildings or structures can pose limits on the direct channel link between a base station (BS) and an Internet of Thing device (IoTD) for wireless communication. Unmanned aerial vehicles (UAVs) with a mounted reconfigurable intelligent surface (RIS), denoted as UAV-RIS, have been introduced in recent works to enhance the system throughput capacity by acting as a relay node between the BS and the IoTDs in wireless access networks. Uncoordinated UAVs or RIS phase shift elements will make unnecessary adjustments that can significantly impact the signal transmission to IoTDs in the area. The concept of Age of Information (AoI) is proposed in wireless network research to categorize the freshness of the received update message. To minimize the Average Sum of AoI (ASoA) in the network, two model-free deep reinforcement learning (DRL) approaches—Off-Policy deep <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i> -network (DQN) and On-Policy proximal policy optimization (PPO)—are developed to solve the problem by jointly optimizing the RIS phase shift, the location of the UAV-RIS, and the IoTD transmission scheduling for large-scale Internet of Things wireless networks. Analysis of loss functions and extensive simulations is performed to compare the stability and convergence performance of the two algorithms. The results reveal the superiority of the On-Policy approach, PPO, over the Off-Policy approach, DQN, in terms of stability, convergence speed, and under diverse environment settings.