Adaptive Optimal Control via Continuous-Time <i>Q</i>-Learning for Stackelberg–Nash Games of Uncertain Nonlinear Systems
Shuhang Yu, Huaguang Zhang, Zhongyang Ming, Jiayue Sun
Abstract
In order to solve the two-player Stackelberg differential game (SDG) for the continuous-time nonlinear Markov jump system (MJS), this article defines a unique <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function and suggests a novel adaptive dynamic programming (ADP) method which is completely independent of system information. First, the optimal policies for the leader and follower are determined from down to the top, and it is further demonstrated that these policies are what make up the Stackelberg–Nash equilibrium point. Then, a novel action-dependent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -function is established in order to attain completely model-free learning, which is the first attempt for SDG-based nonlinear MJS. Furthermore, the Lyapunov direct approach is employed to guarantee the stability of the closed-loop uncertain nonlinear MJS under the control scheme based on ADP, ensuring uniform ultimate boundedness (UUB). Ultimately, a numerical simulation is presented to validate the efficacy of the aforementioned ADP-based control approach.