Constrained Reinforcement Learning-Based Closed-Loop Reference Model for Optimal Tracking Control of Unknown Continuous-Time Systems

Abstract

Although reinforcement learning (RL) is effective in stabilizing systems, it faces many challenges in solving the tracking problem of unknown continuous-time systems. One of the major challenges is that RL-based control can hardly satisfy both the transient and steady-state performance requirements for the tracking problem simultaneously. In this study, instead of implementing an RL controller, the RL agent acts as a planner in the closed-loop reference model. The RL-based planner concentrates on tracking performance optimization by the constrained integral RL algorithm. Meanwhile, the system is controlled by the proposed library-based adaptive controller, which contains a library of candidate functions for modeling the unknown system dynamics. A natural gradient-like adaptive law is developed to update the controller, ensuring asymptotic tracking and promoting sparsity in the controller parameter. Compared with the conventional RL-based control, the proposed framework can eliminate the tracking error while avoiding the high-frequency oscillation and peaking phenomenon. Furthermore, we theoretically demonstrate that our approach can improve the transient performance in terms of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\cal L}_{2} $</tex-math> </inline-formula> norm of the tracking error and explicitly limit the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\cal L}_{\infty} $</tex-math> </inline-formula> norm of the peaking value through the Lyapunov analysis. Simulations are presented to support the theoretical findings at the end of the paper. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Practical control design is often interested in tracking non-zero reference trajectories. However, the non-optimal transient response, such as oscillation and overshoot, is the major obstacle to the development of a high-performance tracking control system. The proposed method addresses this issue by designing a constrained RL-based CRM to ensure the optimal transient performance of the system. The primary advantage is that the maximum peaking value can be conveniently tuned as a hyperparameter by users, which is extremely useful in practice. Furthermore, the proposed library-based adaptive controller can handle unknown system dynamics, where the governing equations of the dynamics can be determined through engineering experience.

Topics & Concepts