Offline Meta-Reinforcement Learning for Active Pantograph Control in High-Speed Railways
Hui Wang, Zhigang Liu, Guiyang Hu, Xufan Wang, Zhiwei Han
Abstract
Previous reinforcement learning (RL) methods suffer significant performance degradation or collapse when deployed to the real world due to the huge sim-real gap. This article proposes a hybrid offline-and-online meta-RL (HOMRL) algorithm that leverages prior task experience to learn and adapt to new pantograph active control tasks in real-world applications. The policy learning process consists of three phases: offline meta-policy pretraining, online adaptation, and fine-tuning. First, we construct an offline meta-RL approach that learns from the massive and heterogeneous static training datasets, eliminating online interaction's high cost and hazard. Second, we combine context-based meta-RL with online fine-tuning to generalize to challenging tasks, while high safety and success rates are critical in railway applications. Finally, the proposed environment-sensitive task encoder (TE) and well-trained agent can adapt to new tasks quickly and efficiently, even in unseen tasks and nonstationary environments. If the new task is similar to the prior data, the contextual meta-learner adapts immediately. If it is too different, it gradually adapts through fine-tuning.