Comprehensive Review of Benefits from the Use of Sparse Updates Techniques in Reinforcement Learning: Experimental Simulations in Complex Action Space Environments
Martin Kaloev, Georgi Krastev
Abstract
This paper presents a comprehensive exploration of the advantages associated with the utilization of sparse update techniques in Artificial Neural Networks (ANNs) within the context of Reinforcement Learning (RL). The study leverages simulations with environments featuring complex action spaces and wide state spaces, characterized by state vectors of dimension 128 and a choice of 9 distinct actions. The research investigates the impact of different update frequencies in ANNs when applied to RL agents in these simulation environments. The experimental methodology entails testing ANN updates after each action executed by the RL agent, followed by evaluations at progressively less frequent intervals. To assess the effectiveness of these techniques, several critical attributes are compared across simulations. These attributes encompass the collected rewards by the RL agents, the number of actions required to accomplish specific tasks within the complex environments, and the computation time necessary to achieve task completion. The study reveals that the optimal update frequency for RL agents using ANNs varies depending on the specific simulation environment. Some environments benefit from more frequent updates, while others perform better with fewer updates. However, excessively sparse updates can lead to performance drops. These findings emphasize the importance of adaptable update schedules tailored to each RL task, which can significantly enhance RL algorithm design in complex environments with wide action and state spaces.