General multi-step value iteration for optimal learning control
Ding Wang, Jiangyu Wang, Derong Liu, Junfei Qiao
Abstract
Learning control methods have been widely enhanced by reinforcement learning, but it is challenging to analyze the effects of incorporating extra system information. This paper presents a novel multi-step framework that utilizes extra multi-step system information to solve optimal control problems. Within this framework, we establish and classify general multi-step value iteration (MsVI) algorithms based on the uniformity between policy evaluation and improvement stages. According to this uniformity concept, the convergence condition and the acceleration conclusion are analyzed for different kinds of MsVI algorithms. Besides, we introduce a swarm policy optimizer to relieve limitations of the traditional gradient optimizer. Specifically, we implement general MsVI using an actor–critic scheme, where the swarm optimizer and neural networks are employed for policy improvement and evaluation, respectively. Furthermore, the approximation error caused by the approximator is also considered to verify the advantage of using multi-step system information. Finally, we apply the proposed method to a nonlinear benchmark system, demonstrating superior learning ability and control performance compared to traditional methods.