Delay-Aware Optimization of Fine-Grained Microservice Deployment and Routing in Edge via Reinforcement Learning
Kai Peng, Jintao He, Jialu Guo, Yuan Liu, Jianwen He, Wei Liu, Menglan Hu
Abstract
Microservices have exerted a profound impact on the development of internet applications. Meanwhile, the growing number of mobile terminal user requests has made the communication between microservices extremely complex, significantly impacting the quality of user service experience in mobile edge computing. Therefore, the joint optimization of microservice deployment and request routing is necessary to alleviate server pressure and enhance overall performance of large-scaled MEC applications. However, most existing work studies the microservice deployment and request routing as two isolated problems and neglects the dependencies between microservices. This paper focuses on the data dependency relationship of request and multi-instance processing problem, and then formulate the joint problem of microservice deployment and request routing as an integer nonlinear program and queuing optimization model under complex constraints. To address this problem, we propose a fine-grained reinforcement learning-based algorithm named Reward Memory Shaping Deep Deterministic Policy Gradient (RMS <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\_$</tex-math></inline-formula> DDPG). Furthermore, we introduce the Long Short-Term Memory (LSTM) block into the actor network and critical network to make actions memorable. Finally, our experiments demonstrate that our algorithm is more superior in terms of delay target, load balancing and algorithm robustness compared with four baseline algorithms.