Deep reinforcement learning-based strategy for maximizing returns from renewable energy and energy storage systems in multi-electricity markets
Javier Cardo-Miota, Héctor Beltrán, Emilio Pérez, Shafi Khadem, Mohamed Bahloul
Abstract
The integration of Renewable Energy Sources (RES) with Energy Storage Systems (ESS) presents challenges and opportunities in optimizing their participation in electricity markets. This study introduces a novel approach that leverages Deep Reinforcement Learning (RL) algorithms to develop optimal bidding strategies for collocated RES with Battery ESS (BESS) configurations, enabling multi-market participation in both energy and ancillary services (AS) markets. The proposed method uses a Markov Decision Process (MDP) framework to manage BESS utilization dynamically, considering market conditions and technical constraints. As an RL agent, the actor–critic approach known as the Twin Delayed Deep Deterministic (TD3) Policy Gradient algorithm is implemented. A data-driven training process facilitates model learning while minimizing the required training dataset and time. Focused on the Irish context, the case study involves participation in both the day-ahead energy market and reserve services for frequency droop curve response of the DS3 Programme. Historical data from a 7 MW solar PV plant and a 1 MWh BESS are utilized to evaluate the performance. The RL agent dynamically adapts to market dynamics and system constraints, achieving substantial economic benefits compared to benchmark strategies, with an additional 8271 € , 166,738 € , and 11,369 € , respectively. • Deep reinforcement learning algorithms for bidding strategies. • Decision-making algorithm for participating in multiple electricity markets. • Operation of renewable energy systems co-located with battery energy storage systems. • Introduces a versatile framework that can be applied universally.