A Model-Free Combined Energy and Thermal Management Strategy for HEVs Based on Reinforcement-Learning Under Low-Temperature
Kai Li, Hong Chen, Yuhu Wu, Jing Zhao, Shihong Ding, Jinwu Gao
Abstract
To improve vehicle adaptability to low-temperature environments, this paper proposes a combined energy and thermal management strategy (C-ETM) based on twin delayed deep deterministic policy gradient (TD3) algorithm for hybrid electric vehicles (HEVs). First, a vehicle energy management system (EMS) model and a engine-battery-cabin coupled thermal management system (CTMS) model are developed. By analyzing the coupling relationship between the CTMS and the EMS, a multi-objective optimization problem is constructed to minimize fuel consumption and battery aging damage and ensure SOC stability. Facing the challenges of solving optimization problems caused by the high-order complex nonlinearity of thermal-electrical coupling systems, the optimization problems are transformed into a Markov decision process (MDP). A reinforcement learning framework based on the TD3 algorithm is designed to achieve a real-time solution to the problem from a new perspective, overcoming the reliance on the system models and accurate future traffic information. The proposed strategy has efficient performance in terms of fuel economy, battery life, ensuring SOC stability, and adaptability. The total optimization cost reaches 91.42<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> level of the dynamic programming (DP) strategy, which is 30.3<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> lower than the model predictive control (MPC) strategy. The online computing burden is only 0.19<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> of the MPC strategy, which has strong potential for real-time applications.