Exploiting Partial Observability and Optimized Simple State Representations in Deep Q-Learning
Danyaal Mahmood, Usama Arshad, Raja Hashim Ali, Zain Ul Abideen, Muhammad Huzaifa Shah, Talha Ali Khan, Ali Zeeshan Ijaz, Nisar Ali, Abu Bakar Siddique
Abstract
The advent of deep Q-learning has opened up new possibilities in training autonomous agents to perform intelligently in intricate settings. This research work examines the potential of deep Q-learning in the paradigmatic Snake game, which requires an agent to navigate a grid, ingest food items, and avoid collisions. We incorporated a partially observable game state, introducing a novel level of difficulty to the task. Our proposed approach utilizes a straightforward feed-forward neural network to estimate Q-values of potential actions, given the current state. The agent progressively learns to balance between exploration and exploitation, allowing it to navigate optimally in the game scenario. Most existing studies in deep Q-learning for the Snake game overlook the complexities posed by partial observability, focusing mainly on fully observable state representations. Furthermore, the majority of the investigations delve into the efficacy of complex neural network architectures, leaving a gap in the understanding of the performance of more simplistic feed-forward networks. Our study fills this gap by offering an in-depth exploration and analysis of deep Q-learning strategies, especially when applied to partially observable game states and simple neural networks. Simulations and results reveal that adopting simpler state representations can significantly boost the agent’s performance in the Snake game. A more detailed and intricate reward function is also observed to enhance the agent’s decision-making skills and overall proficiency in the game. These observations underscore the significance of optimizing state representations and reward functions for maximizing the performance of deep Q-learning agents in the Snake game.