QoE-Driven Adaptive Deployment Strategy of Multi-UAV Networks Based on Hybrid Deep Reinforcement Learning
Yi Zhou, Xiaoyong Ma, Shuting Hu, Danyang Zhou, Nan Cheng, Ning Lu
Abstract
Unmanned aerial vehicles (UAVs) serve as aerial base stations to provide controlled wireless connections for ground users. Due to their constraints on both mobility and energy consumption, a key problem is how to deploy UAVs adaptively in a geographic area with changing traffic demand of mobile users, while meeting the aforementioned constraints. In this article, we propose a Quality of Experience (QoE)-driven and energy-efficient adaptive deployment strategy for multi-UAV networks based on hybrid deep reinforcement learning (DRL) to solve the problem of incomplete information game, where the UAVs can adjust their moving directions and distance to serve users who move randomly in the target area. Through the hybrid DRL with centralized training and distributed testing, UAVs can be trained offline to obtain the global state information and learn a completely distributed control strategy, with which each UAV only needs to take actions based on its observed state in the real deployment to be fully adaptive. Moreover, in order to improve the speed and effect of learning, we improve hybrid reinforcement learning, by adding genetic algorithms and temporal difference error-based resampling optimization mechanism. The simulation results show that the hybrid DRL algorithm has better efficiency and robustness in multi-UAV control, and has better performance in terms of QoE, energy consumption, and average throughput, by which average throughput can be increased by 20%–60%.