<i>Q</i> Learning-Based Routing Protocol With Accelerating Convergence for Underwater Wireless Sensor Networks
Chao Wang, Xiaohong Shen, Haiyan Wang, Weiliang Xie, Haodi Mei, Hongwei Zhang
Abstract
Underwater wireless sensor networks (UWSNs) have emerged as a promising technology for various underwater applications. Considering the characteristics such as limited energy and high end-to-end delay in UWSNs, it is important to design an underwater routing protocol with high energy efficiency, low end-to-end delay, and high reliability. Therefore, a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> learning (QL)-based routing protocol is proposed in this article. First, a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> learning-based framework is constructed by considering link connectivity and the information of residual energy, depth, and neighboring nodes. The framework enables protocols to adapt to the dynamic environment and facilitate efficient transmission. Furthermore, to address the slow convergence of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> learning in UWSNs, a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> value initialization strategy using layer information is designed to accelerate the convergence speed. In addition, an adaptive discount mechanism and a dynamic learning mechanism are proposed to update <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> values for adapting to the changing network topology and improve the reliability of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> values for nodes rarely selected, respectively. Finally, the superior performance of the proposed protocol is evaluated through simulations. Simulation results show that the proposed protocol can still accelerate the convergence speed in reducing the energy tax by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${37}.{16}\%$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${23}.{08}\%$ </tex-math></inline-formula> , and the average end-to-end delay by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${29}.{94}\%$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${16}.{91}\%$ </tex-math></inline-formula> as compared to other <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> learning-based routing protocols QELAR and QDAR under dynamic environment, while maintaining a higher packet delivery ratio (PDR).