Transfer Reinforcement Learning for Dynamic Spectrum Environment
Hanmin Sheng, Wenjian Zhou, Jiajun Zheng, Yuan Zhao, Wenjian Ma
Abstract
Reinforcement learning (RL) has proven to be an effective approach for achieving intelligence in Cognitive Radio (CR). Through interactions with the environment, RL enables a CR to optimize in an efficient and flexible manner. The vast majority of studies, however, are carried out in a spectrum environment with prefixed user access rules, typically with a constant transition probability and reward distribution. In fact, in a real-world spectrum environment, changes in access rules are common, which has a significant impact on the effectiveness of RL, while few studies have been conducted on this topic. This paper demonstrates how changes in primary user’s (PU) access rules affect RL strategies. To improve the secondary user’s (SU) performance for the dynamic spectrum environment, a transfer Deep Q-Network (DQN) is proposed, this method screens out knowledge from historical experience while avoiding interference from irrelevant information with an experience playback mechanism. Experiments show that this method outperforms traditional RL methods in terms of conflict rate, spectrum utilization rate, and convergence rate in the dynamic spectrum. Given the scarcity of studies on this topic, this study is expected to serve as a benchmark for the future research.