Resilient Control of Networked Microgrids Using Vertical Federated Reinforcement Learning: Designs and Real-Time Test-Bed Validations
Sayak Mukherjee, Ramij Raja Hossain, Sheik M. Mohiuddin, Yuan Liu, Wei Du, Veronica Adetola, Rohit Jinsiwale, Qiuhua Huang, Tianzhixi Yin, Ankit Singhal
Abstract
Improving system-level resiliency of networked microgrids against adversarial cyber-attacks is an important aspect in the current regime of increased inverter-based resources (IBRs). To achieve that, this paper contributes in designing a hierarchical control layer, in conjunction with the existing control layers, resilient to adversarial attack signals. Considering model complexities, unknown dynamical behaviors of IBRs, and privacy issues regarding data sharing in multi-party-owned microgrids, designing such a control layer is non-trivial. Here, to tackle these issues, a novel federated reinforcement learning (Fed-RL) method is proposed. To grasp the interconnected dynamics of networked microgrids, the paper develops Federated Soft Actor-Critic (FedSAC) algorithm following the vertical structure of implementing Fed-RL. Next, utilizing the OpenAI Gym interface, we built a custom set-up in GridLAB-D/HELICS co-simulation platform, named Resilient RL Co-simulation (ResRLCoSIM), to train the RL agents with IEEE 123-bus benchmark comprising 3 interconnected microgrids. Finally, the learned policies in the simulation are transferred to the real-time hardware-in-the-loop (HIL) test-bed developed using the high-fidelity Hypersim platform. Experiments show that the simulator-trained RL controllers achieve desirable performance with the test-bed platform, validating the minimization of the sim-to-real gap.