Multi-Channel Opportunistic Access for Heterogeneous Networks Based on Deep Reinforcement Learning
Xiaowen Ye, Yiding Yu, Liqun Fu
Abstract
This paper investigates a new medium access control (MAC) protocol for multi-channel heterogeneous networks (HetNets) based on deep reinforcement learning (DRL), referred to as multi-channel deep-reinforcement learning multiple access (MC-DLMA). Specifically, we consider a HetNet where different radio networks adopt different MAC protocols to transmit data packets to a common access point on different wireless channels. Three key challenges for the MC-DLMA node are (i) no environmental knowledge is known in advance; (ii) the channels in HetNets are allocated to nodes using different MAC protocols; (iii) the capacities of different channels may be different. The main goal of MC-DLMA is to find an optimal access policy to transmit on those pre-allocated channels and expedite more efficient spectrum utilization. Due to the complex temporal correlation of spectrum states in HetNets, the traditional DRL technique, e.g., original deep Q-network (DQN) algorithm, is no longer applicable to our problem. In our MC-DLMA design, an advanced class of recurrent neural network, termed as Gated Recurrent Unit (GRU), is embedded into the original DQN technique to aggregate observations over time and reason the underlying temporal feature in multi-channel HetNets. Furthermore, we analytically give the optimal spectrum access patterns and derive the optimal throughputs in various HetNet scenarios. With judicious definitions of the state, action, and reward function in the parlance of the DRL framework, simulation results show that MC-DLMA can (i) find the optimal spectrum access strategies in various HetNets, (ii) outperform the random access policy, the whittle index policy, and the original DQN, (iii) perform cooperative transmission in a fully distributed manner in the presence of multiple agents, and (iv) adapt well to the environmental changes.