Online-Learning Adaptive Energy Management for Hybrid Electric Vehicles in Various Driving Scenarios Based on Dyna Framework
Ningkang Yang, Lijin Han, Xuan Zhou, Rui Liu, Hui Liu, Changle Xiang
Abstract
The practical driving scenarios have a decisive influence on the performance of energy management strategies (EMSs) for hybrid electric vehicles (HEVs). How to adapt to various scenarios is still a challenging task for reinforcement learning (RL) based EMSs. To tackle this problem, the paper proposes an online-learning adaptive EMS based on a RL framework, Dyna. In the framework, besides directly improving the policy, the real experience from the vehicle is reused to establish an interactive model. Through the model, simulated experience is generated to meanwhile improve the policy and most high-priced real-world exploration is substituted by model-based exploration, which substantially reduces the learning time and lowers the learning price. Thus, Dyna-based EMS achieves fast and low-cost online learning when the HEV enters a new scenario, significantly enhancing the strategy’s adaptability. In the simulation, three typical driving scenarios are first selected for testing: city, suburb and highway. For all three scenarios, Dyna only cumulates about 60% driving cost of DQN in the first 100 cycles. In an off-road scenario, Dyna also achieves about 70% cumulative costs compared with DDPG and TD3 considering 100 cycles, which demonstrates the superiority in the adaptability of the proposed EMS.