Local Observations-Based Energy-Efficient Multi-Cell Beamforming via Multi-Agent Reinforcement Learning
Kaiwen Yu, Gang Wu, Shaoqian Li, Geoffrey Ye Li
Abstract
With affordable overhead on information exchange, energy-efficient beamforming has potential to achieve both low power consumption and high spectral efficiency. This paper formulates the problem of joint beamforming and power allocation for a multiple-input single-output (MISO) multi-cell network with local observations by taking the energy efficiency into account. To reduce the complexity of joint processing of received signals in presence of a large number of base station (BS), a new distributed framework is proposed for beamforming with multi-cell cooperation or competition. The optimization problem is modeled as a partially observable Markov decision process (POMDP) and is solved by a distributed multi-agent self-decision beamforming (DMAB) algorithm based on the distributed deep recurrent Q-network (D<sup>2</sup>RQN). Furthermore, limited-information exchange scheme is designed for the inter-cell cooperation to boost the global performance. The proposed learning architecture, with considerably less information exchange, is effective and scalable for a high-dimensional problem with increasing BSs. Also, the proposed DMAB algorithms outperform distributed deep Q-network (DQN) based methods and non-learning based methods with significant performance improvement.