Multi-USV cooperative target encirclement through learning-based distributed transferable policy and experimental validation
Chenming Zhang, Rijie Zeng, Bin Lin, Yibo Zhang, Wei Xie, Weidong Zhang
Abstract
Coordinated strategies for unmanned surface vehicles (USVs) to encircle an escaping target is a key research focus in maritime operations. In this paper, we propose a distributed, transferable multi-agent reinforcement learning (MARL) algorithm to address this challenge. The algorithm achieves the target encirclement for multi-agent swarm based on limited observational information, while there are random obstacles in the scenario. A centralized training decentralized execution (CTDE) deep reinforcement learning (DRL) framework is introduced. Also, the framework facilitates transfer of policies to larger swarms. To balance individual and collective benefits, we propose a knowledge-embedded compound reward function. Nonlinear mapping terms are added to the reward function to meet the requirements of the encirclement task at different stages. Furthermore, to improve training effectiveness, we implement a parallel training method and a multi-agent policy-sharing (MAPS) mechanism for isomorphic agents. Multiple tests indicate the effectiveness of the proposed algorithm. We developed a framework for deploying the learning-based policies in real-world scenarios. The MARL-based algorithm proposed in this paper has been successfully deployed in an indoor pool.