Adaptive Control Scheme for USV Trajectory Tracking Under Complex Environmental Disturbances via Deep Reinforcement Learning
Yuan Zhou, Chongwei Gong, Keran Chen
Abstract
Unmanned surface vehicles (USVs) have demonstrated impressive practical value and potential in Marine Internet of Things (MIoT) system. Although trajectory-tracking control is among the most common practical technology of USVs, various limitations remain unaddressed. Existing studies have employed simple mathematical models to simulate marine environment without utilizing actual data, resulting in a lack of environmental authenticity. Moreover, a complex marine environment increases the need for robustness and adaptability of the control policy. To overcome these limitations, this study proposes a deep reinforcement learning (DRL)-based policy for USV trajectory-tracking control, which can effectively adapt to complex environmental disturbances. First, we use actual marine data, including ocean currents and winds, to construct a time-varying multi-element marine environment model. Next, an effective Markov decision processes (MDPs) formulation integrating LOS guidance law is elaborately proposed, in which the composite reward function and state transition function are used to avoid ineffective exploration and achieve better convergence ability. Furthermore, a USV trajectory-tracking controller based on hybrid priority twin-delayed deep deterministic policy gradient (TD3) agent is designed; specifically, a hybrid priority experience replay mechanism is developed and integrated within the TD3. It evaluates the significance of an experience by weighing the temporal-difference (TD) error and reward value, thus enabling the USV agent to explore optimal control policies and further accelerate the network convergence. Experimental results show that our method achieves better trajectory-tracking performance than mainstream DRL-based and model-based control approaches, and adapts to different reference trajectories and different intensities of environmental disturbances with high tracking accuracy.