A scalable approach for real-world implementation of deep reinforcement learning controllers in buildings based on online transfer learning: The HiLo case study
Davide Coraci, Alberto Silvestri, Giuseppe Razzano, Davide Fop, Silvio Brandi, Esther Borkowski, Tianzhen Hong, Arno Schlueter, Alfonso Capozzoli
Abstract
In recent years, Transfer Learning (TL) has emerged as a promising solution to scale Deep Reinforcement Learning (DRL) controllers for building energy management, addressing challenges related to DRL implementation as high data requirements and reliance on surrogate models. Moreover, most TL applications are limited to simulations, not revealing their real performance in actual buildings. This paper explores the implementation of an online TL methodology combining imitation learning and fine-tuning to transfer a DRL controller between two real office environments. Pre-trained in simulation using a calibrated digital twin, the DRL controller reduces energy consumption and improves indoor temperature control when managing the operation of a Thermally Activated Building System in one of the two offices both in simulation and in the real field. Afterwards, the DRL controller is transferred to the other office following the online TL methodology. The proposed approach outperforms a DRL controller implemented without pre-training, and Rule-Based and Proportional-Integral controllers, achieving energy savings between 6 and 40% and improving indoor temperature control between 30 and 50%. These findings underscore the efficacy of the online TL methodology as a viable solution to enhance the scalability of DRL controllers in real buildings. • Real implementation of an online transfer learning strategy for DRL controllers. • DRL controller pre-trained in simulation before deployed in the real source office. • Performance benchmarking of online transfer learning by employing digital twins. • Online transfer learning outperforms RBCs, PI and online DRL controllers.