Deep Reinforcement Learning design of safe, stable and robust control for sloshing-affected space launch vehicles
Périclès Cocaul, Sylvain Bertrand, Hélène Piet-Lahanier, Lori Lemazurier, Martine Ganet
Abstract
New challenges in spatial missions and the design of new launchers entail a focus on innovative control strategies. Recent developments in Machine Learning (ML) for optimization processes shed light on the possibilities offered for controlling complex nonlinear partially unknown systems. This work focuses on the use of these methods to design control laws stabilizing the sloshing of propellants in tanks during launcher flight. A major hurdle in applying control laws designed by Artificial Intelligence (AI) to safety-critical systems lies in certifying stability and safety. Using Control Lyapunov Function (CLF) and Control Barrier Function (CBF) developed in Control Theory approaches, closed-loop stability and safety in terms of state constraints can be verified. Considering a Deep Reinforcement Learning (DRL) framework, an algorithm is developed to learn a control policy along with stability and safety certificates. The CLF and CBF conditions are integrated in the DRL algorithm, bridging the gap between Control Theory and Machine Learning techniques. A safe and stable DRL controller is then learned on a simulated launcher subject to sloshing with uncertainties and perturbations due to sloshing. A robustness study with Monte Carlo simulations is conducted to evaluate performance under various conditions. Finally, the developed controller is validated on an industrial simulator that more accurately models the real behavior of the launcher. Despite not being trained on this industrial simulator, the controller matches control objectives, demonstrating robustness and performance. • Develop a safe and stable model-free DRL controller for a launcher. • Ensure closed-loop stability and satisfaction of state and command constraints. • Assess the learned controller’s robustness with respect to uncertainties and delays. • Apply the learned controller on an industrial safety-critical case study.