Frames-of-Reference-Based Learning: Overcoming Perceptual Aliasing in Multistep Decision-Making Tasks
Abubakar Siddique, Will N. Browne, Gina M. Grimshaw
Abstract
Perceptual aliasing challenges reinforcement learning agents. They struggle to learn stable policies by failing to identify and disambiguate perceptually identical states in the environment that require different actions to reach a goal. As the agent often has only a local frame of reference, it cannot represent the global environment. Frame-of-reference-based learning is a feature of vertebrate intelligence that allows multiple simultaneous representations of an environment at different levels of abstraction. This enables the resolution of patterns that are made up of patterns that are made up of features. The evolutionary computation technique of learning classifier systems has shown promise in learning nested patterns in single-step domains. This work uses the frame-of-reference concept within a learning classifier system to learn stable policies in non-Markov multistep domains. Considering aliased states at a constituent level enables the system to place them appropriately in holistic-level policies. Instead of enumerating a huge search space, evolution computation empowers the novel system to evolve fitter rules and policies. The experimental results show that the novel system effectively solves complex aliasing patterns in non-Markov environments that have been challenging to artificial agents. For example, the novel system utilizes only 6.5, 3.71, and 3.22 steps to resolve Maze10, Littman57, and Woods102, respectively.