Leveraging Deep Reinforcement Learning for Water Distribution Systems with Large Action Spaces and Uncertainties: DRL-EPANET for Pressure Control
Anas Belfadil, David Modesto, Jordi Meseguer, Bernat Joseph‐Duran, D. Saporta, José Antonio Martín Hernández
Abstract
Deep reinforcement learning (DRL) has undergone a revolution in recent years, enabling researchers to tackle a variety of previously inaccessible sequential decision problems. However, its application to the control of water distribution systems (WDS) remains limited. This research demonstrates the successful application of DRL for pressure control in WDS by simulating an environment using EPANET version 2.2, a popular open-source hydraulic simulator. We highlight the ability of DRL-EPANET to handle large action spaces, with more than 1 million possible actions in each time step, and its capacity to deal with uncertainties such as random pipe breaks. We employ the Branching Dueling Q-Network (BDQ) algorithm, which can learn in this context, and enhance it with an algorithmic modification called BDQ with fixed actions (BDQF) that achieves better rewards, especially when manipulated actions are sparse. The proposed methodology was validated using the hydraulic models of 10 real WDS, one of which integrated transmission and distribution systems operated by Hidralia, and the rest of which were operated by Aigües de Barcelona.