Optimal Control for Continuous-Time Unknown Nonlinear Affine Systems: A <i>Q</i>-Learning Approach
Shuhang Yu, Huaguang Zhang, Zhongyang Ming, Jiayue Sun
Abstract
In this paper, to tackle the optimal control problem, we propose a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -Learning approach for continuous-time nonlinear systems without any dynamic information. Primarily, the Hamiltonian and optimum cost functions are utilized to articulate the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -function of continuous-time affine systems. To reduce the dependence of algorithms on system information, a novel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -Learning approach is derived to obtain optimal solutions of nonlinear continuous-time systems without requiring knowledge of either the drift information <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$p(x)$</tex-math> </inline-formula> or input gain <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$q(x)$</tex-math> </inline-formula> . To implement this approach, critic and actor neural networks can be iterated alternately using an integral reinforcement learning method to estimate the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -function. Furthermore, all signals in closed-loop system are demonstrated to be ultimate uniform bounded (UUB). It is worth noting that there exist rare literatures focused on the optimal control problem of continuous-time nonlinear uncertain systems via the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -Learning for actor/critic networks iteration. Finally, two simulations are used to confirm the effectiveness of the proposed algorithm. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Nonlinear continuous-time systems, being ubiquitous in engineering practice, are widely employed due to their versatility and effectiveness. Aiming at such systems, a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -learning approach with optimal feature is proposed to strengthen control efficiency while reduce costs. However, it is well known that accurately capturing all the dynamic information of the system is a formidable task in practical operation. This defect inevitably weakens the feasibility of model-based control algorithms. Since the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal{Q}$</tex-math> </inline-formula> -learning algorithm presented in this paper does not require any dynamic knowledge of systems, it is promising enabler in enhancing the effectiveness and flexibility of engineering activities.