Improved Reinforcement Learning Using Stability Augmentation With Application to Quadrotor Attitude Control
Hangxing. Wu, Hui Ye, Wentao Xue, Xiaofei Yang
Abstract
Reinforcement learning (RL) has been successfully applied to motion control, without requiring accurate models and selection of control parameters. In this paper, we propose a novel RL algorithm based on proximal policy optimization algorithm with dimension-wise clipping (PPO-DWC) for attitude control of quadrotor. Firstly, dimension-wise clipping technique is introduced to solve the zero-gradient problem of the PPO algorithm, which can quickly converge while maintaining good sampling efficiency, thus improving the control performance. Moreover, following the idea of stability augmentation system (SAS), a feedback controller is designed and integrated into the environment before training the PPO controller to avoid ineffective exploration and improve the system’s convergence. The eventual controller consists of two parts: the first is the result of the actor neural network in the PPO algorithm, and the second is the output of the stability augmentation feedback controller. Both of them directly use an end-to-end style of control commands to map the system state. This control architecture is applied in the attitude control of the quadrotor. The simulation results show that the quadrotor can quickly and accurately track the command and has a small steady-state error after the training by the improved PPO algorithm. Meanwhile, compared with the traditional PID controller and basic PPO algorithm, the proposed PPO-DWC algorithm with stability augmentation framework has better performance in tracking accuracy and robustness.