Model-Free Control in Wireless Cyber–Physical System With Communication Latency: A DRL Method With Improved Experience Replay
Yifei Qiu, Shaohua Wu, Jian Jiao, Ning Zhang, Qinyu Zhang
Abstract
This article explores the model-free remote control problem in a wireless networked cyber-physical system (CPS) composed of spatially distributed sensors, controllers, and actuators. The sensors sample the states of the controlled system to generate control instructions at the remote controller, while the actuators maintain the system's stability by executing control commands. To realize the control under a model-free system, the deep deterministic policy gradient (DDPG) algorithm is adopted in the controller to enable model-free control. Unlike the traditional DDPG algorithm, which only takes the system state as input, this article incorporates historical action information as input to extract more information and achieve precise control in the case of communication latency. Additionally, in the experience replay mechanism of the DDPG algorithm, we incorporate the reward into the prioritized experience replay (PER) approach. According to the simulation results, the proposed sampling policy improves the convergence rate by determining the sampling probability of transitions based on the joint consideration of temporal difference (TD) error and reward.