Lexicographic Actor-Critic Deep Reinforcement Learning for Urban Autonomous Driving
Hengrui Zhang, Youfang Lin, Sheng Han, Kai Lv
Abstract
Urban autonomous driving is a difficult task because of its complex road scenarios and the interaction between multiple vehicles. Autonomous vehicles need to balance multiple objectives in these complex scenarios, e.g., safety and speed. Traditional reinforcement learning methods deal with the multi-objective problem by optimizing agents with a single objective reward. However, these methods are sensitive to the reward scale and require huge experiments to design reward weights. In this paper, we propose the Lexicographical Proximal Policy Optimization algorithm (LPPO), which can express people's preference relationship through the lexicographical ordering between objectives. The proposed method has two main advantages. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">On the one hand,</i> LPPO has a smaller parameter adjustment space, which makes it easy to find the optimal solution that satisfies the actual problem preference. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">On the other hand,</i> the proposed method is less affected by the reward scale and easy to deploy in various driving scenarios. We evaluate our algorithm in two driving simulation environments, and the results show that the proposed method has better performance in urban driving tasks than previous reinforcement learning algorithms. In addition, we illustrate that the proposed method has better stability even if the reward scale changes.