Hierarchical Safe Reinforcement Learning Control for Leader-Follower Systems With Prescribed Performance
Junkai Tan, Shuangsi Xue, Huan Li, Zihang Guo, Hui Cao, Badong Chen
Abstract
This paper proposes a hierarchical safe reinforcement learning with prescribed performance control (HSRL-PPC) scheme to address the challenges of interconnected leader-follower systems operating in complex environments. The framework consists of two levels: at the higher level, the leader agent detects and avoids moving obstacles while planning optimal paths; at the lower level, the follower agent tracks the leader within strict prescribed performance bounds. We formulate the optimal prescribed performance safe control problem and solve it using the Hamilton-Jacobi-Bellman (HJB) equation. Due to system nonlinearity and obstacle complexity, we approximate the leader’s optimal value function using a state-following neural network that efficiently extrapolates training data to neighboring states, while employing a regular critic neural network for the follower’s value function approximation. Lyapunov stability analysis demonstrates the closed-loop system’s theoretical guarantees. Experimental results from two simulation examples and hardware tests with a quadcopter-vehicle system validate the effectiveness of the proposed approach in achieving safe navigation and precise tracking performance in dynamic environments. Note to Practitioners—Challenges exist in unpredictable obstacles and agent limitations for the interconnected leader-follower system. To provide a safe, efficient, and reliable control scheme, hierarchical safe reinforcement learning with prescribed performance control is proposed in this paper. The hierarchical structure is utilized to coordinate the leader and follower agents in the interconnected system, where the leader agent plans the optimal path and avoids obstacles, and the follower agent tracks the leader within prescribed performance bounds. Based on the proposed hierarchical structure, engineers can design efficient and safe control schemes for interconnected leader-follower systems with moving obstacles. In future work, we will address the problem of external disturbances and uncertainties in the interconnected leader-follower system.