Real-world deployment of model-free reinforcement learning for energy control in district heating systems: Enhancing flexibility across neighboring buildings
Amirhosein Moshari, Kavan Javanroodi, Vahid M. Nik
Abstract
Energy Management Systems (EMSs) often operate through inflexible, rule-based control systems. Model-free reinforcement learning (RL) has emerged as a promising alternative, providing adaptive and autonomous control without the need for detailed modeling. However, the complexity of operating energy systems and dynamic environmental conditions limits their practical use, with most studies relying on simulations or short-term trials that fall short of real-world deployment. This research presents a prolonged, multi-building implementation of a complete, autonomous, model-free RL system that is developed and deployed within operational buildings throughout a full heating season. The system operated successfully without the need for simulation, pre-training, calibration, or additional sensor requirements, and, via flexibility signals, it facilitated a more efficient and robust deployment, while also addressing data privacy concerns. The analysis provided significant results, including a 7.9% decrease from the previous warmer year and a 29.7% reduction in heating energy compared to a multi-year historical baseline. The optimal performance of the implemented system became evident through a 3.85 °C reduction in return temperature and consistent reductions in peak demand, while maintaining occupant comfort. By combining ANCOVA normalization, matched-temperature baselines, thermodynamic efficiency metrics, and comfort analyses, this study introduces a structured framework and evaluation metrics to support robust field assessments of RL-based control in buildings. Finally, the study highlights the strong potential of autonomous RL engines for broader implementation in complex real-world energy management systems, as they require no knowledge of system dynamics or infrastructure upgrades, offering a reliable and cost-effective solution. • A model-free RL control system was deployed in 13 real buildings for 138 winter days. • No simulation, calibration, or hardware upgrades were needed for RL integration. • Heating energy use dropped by 29.7% versus historical multi-year baselines. • Return water temperatures were lowered by 3.85 °C, improving system efficiency. • RL maintained occupant comfort while shaving thermal peak demand.