Action-Dependent Heuristic Dynamic Programming With Experience Replay for Wastewater Treatment Processes
Junfei Qiao, Mingming Zhao, Ding Wang, Menghua Li
Abstract
The wastewater treatment process (WWTP) is beneficial for maintaining sufficient water resources and recycling wastewater. A crucial link of WWTP is to ensure that the dissolved oxygen (DO) concentration is continuously maintained at the predetermined value, which can actually be considered as a tracking problem. In this article, an experience replay-based action-dependent heuristic dynamic programming (ER-ADHDP) method is developed to design the model-free tracking controller to accomplish the tracking goal of the DO concentration. First, the online ER-ADHDP controller is regarded as a supplementary controller to conduct the model-free tracking control alongside a stabilizing controller with a priori knowledge. The online ER-ADHDP method can adaptively adjust weight parameters of critic and action networks, thereby continuously ameliorating the tracking result over time. Second, the ER technique is integrated into the critic and action networks to promote the data utilization efficiency and accelerate the learning process. Third, a rational stability result is provided to theoretically ensure the usefulness of the ER-ADHDP tracking design. Finally, simulation experiments including different reference trajectories are conducted to show the superb tracking performance and excellent adaptability of the proposed ER-ADHDP method.