Safe Building HVAC Control via Batch Reinforcement Learning
Chi Zhang, Sanmukh R. Kuppannagari, Viktor K. Prasanna
Abstract
In this paper, we study safe building HVAC control via batch reinforcement learning. Random exploration in building HVAC control is infeasible due to safety considerations. However, diverse states are necessary for RL algorithms to learn useful policies. To enable <i>safety</i> during exploration, we propose guided exploration by adding a Gaussian noise to a hand-crafted rule-based controller. Adjusting the variance of the noise provides a tradeoff between the <i>diversity</i> of the dataset and the <i>safety</i> . We apply Conservative Q Learning (CQL) to learn a policy. CQL ensures that the trained policy stays within the policy distribution used to collect the dataset, thereby guarantees safety at deployment. To select the optimal policy during the offline training, we apply model-based performance evaluation. We use the widely adopted CityLearn testbed to evaluate the performance of our proposed method. Compared with a rule-based controller, our approach obtains <inline-formula><tex-math notation="LaTeX">$12\%\sim 35\%$</tex-math></inline-formula> reduction in ramping, <inline-formula><tex-math notation="LaTeX">$3\%\sim 10\%$</tex-math></inline-formula> reduction in 1-load factor, <inline-formula><tex-math notation="LaTeX">$3\%\sim 8\%$</tex-math></inline-formula> reduction in daily peak at deployment with less than <inline-formula><tex-math notation="LaTeX">$10\%$</tex-math></inline-formula> performance degradation during the exploration. On the contrary, the performance degradation of the state-of-the-art online reinforcement learning algorithm during exploration is around <inline-formula><tex-math notation="LaTeX">$8\%\sim 18\%$</tex-math></inline-formula> . It also fails to surpass the performance of the rule-based controller at deployment.