Soft Contrastive Learning With Q-Irrelevance Abstraction for Reinforcement Learning
Minsong Liu, Luntong Li, Shuai Hao, Yuanheng Zhu, Dongbin Zhao
Abstract
The difference between training and testing environments is a huge challenge to generalizing reinforcement learning (RL) algorithms. We propose a soft contrastive learning with a coarser approximate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -irrelevance abstraction for RL (SCQRL) to increase RL generalization. Specifically, we specify the coarser approximate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -irrelevance abstraction as the feature of the state with a theoretical analysis for better generalization ability. We construct a positive and negative sample selection mechanism based on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> value for contrastive learning to achieve efficient representation learning. Considering the selection error of positive and negative samples, we design soft contrastive learning and combine it with RL in the form of an auxiliary task to propose SCQRL. The generalization experiments on several Procgen environments demonstrate that SCQRL outperforms the excellent generalized RL algorithm.