Joint Trajectory and Communication Optimization for Heterogeneous Vehicles in Maritime SAR: Multi-Agent Reinforcement Learning
Chengjia Lei, Shaohua Wu, Yi Yang, Jiayin Xue, Qinyu Zhang
Abstract
Nowadays, multiple types of equipment, including unmanned aerial vehicles (UAVs) and automatic surface vehicles (ASVs), have been deployed in maritime search and rescue (SAR). However, due to the lack of base stations (BSs), how to complete the rescue while maintaining the communication between vehicles is an unresolved challenge. In this paper, we design an efficient and fault-tolerant communication solution by jointly optimizing vehicles' trajectory, offloading scheduling, and routing topology for a heterogeneous vehicles system. First, we model several essential factors in maritime SAR, including the impact of ocean currents, the observational behavior of UAVs, the fault tolerance of relay networks, resource management of mobile edge computing (MEC), and energy consumption. A multi-objective optimization problem is formulated, aiming at minimizing time and energy consumption while increasing the fault tolerance of relay networks. Then, we transfer the objective into a decentralized partially observable Markov Decision Process (Dec-POMDP) and introduce multi-agent reinforcement learning (MARL) to search for a collaborative strategy. Specifically, two MARL approaches with different training styles are evaluated, and three techniques are added for improving performance, including <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sharing parameters</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">normalized generalized-advantage-estimation</i> (GAE), and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">preserving-outputs-precisely-while-adaptively-rescaling-targets</i> (Pop-Art). Experimental results demonstrate that our proposed approach, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">heterogeneous vehicles multi-agent proximal policy optimization</i> (HVMAPPO), outperforms other baselines in efficiency and fault tolerance of communication.