Optimal Energy Management of Plug-In Hybrid Electric Vehicles Through Ensemble Reinforcement Learning With Exploration-to-Exploitation Ratio Control
Bin Shuai, Min Hua, Yanfei Li, Shijin Shuai, Hongming Xu, Quan Zhou
Abstract
Reinforcement learning (RL) has demonstrated its advantages in the intelligent control of many vehicle systems. However, controlling the exploration-to-exploitation (E2E) ratio of RL for the best performance in real-world operations is a great challenge. To obtain the optimal E2E ratio for managing the energy flow of a plug-in hybrid electric vehicle (PHEV) in real-world driving, this paper proposes an ensemble learning scheme based on two independent Q-learning agents working back-to-back competitively. At each sampling time, these agents generate two candidate control actions based on state observation and their control policies. Three decay functions, including the widely-used exponential decay and two new decay methods i.e., reciprocal function-base decay and step-based decay, are introduced to formulate one-dimensional look-up tables for E2E control. Then the PHEV control action will be selected from the candidate actions by a learning automata module(LAM) developed in this paper. The combinations of the three decay methods and three ensemble strategies with maximum-based, randombased, and weighted-based methods are investigated with the considerations of their energy efficiency, real-time performance, and robustness. Experiments are carried out on software-in-loop and hardware-in-the-loop platforms to demonstrate the energysaving potentials. By implementing the ensemble learning scheme based on the weighted-based ensemble method in the control of the studied PHEV, up to 1.04% of energy can be saved under the predefined real-world driving cycles compared to the conventional Q-learning scheme.