Deep Reinforcement Learning Based Dynamic Beam Selection in Dual-Band Communication Systems
Zhen Zhang, Jianhua Zhang, Yuxiang Zhang, Li Yu, Feifei Gao, Qingjiang Shi, Guangyi Liu, Zhiqiang Yuan, Wei Fan
Abstract
To reduce the downlink beam sweep overhead of mmWave systems, we propose a deep reinforcement learning based dynamic beam selection (DRL-DBS) method. A new learning motivation is presented by analyzing the dynamic change laws of high- and low-frequency channels in the spatial domain: to learn the index offset between the optimal beam of mmWave and sub-6 GHz spatial spectrum. In the DRL-DBS method, we propose a novel action space where actions can dynamically adjust the size of the beam sweep subset according to the high-and low-frequency channel propagation laws. Hence, the DRL-DBS method can predict a mmWave downlink beam sweep subset with dynamic size, and the optimal beamforming index is from beam sweep results on the subset. A dual-input dueling Q-network with noisy networks and prioritized experience replay is designed to select the optimal action. The DRL-DBS method can achieve a dynamic trade-off between mmWave beam selection quality and beam sweep overhead based on the reward function. Simulation results demonstrate the superior performance of the DRL-DBS method compared with the existing strategies. Especially, the DRL-DBS method outperforms the exhaustive search algorithm in achievable rate because the overhead of mmWave beam sweep is considered.