REAL: Resilience and Adaptation using Large Language Models on Autonomous Aerial Robots

Abstract

Large Language Models (LLMs) pre-trained on internet-scale datasets have shown impressive capabilities in code understanding, synthesis and in processing extended sequences of symbols, often presented in natural language. This work aims to explore new opportunities in long-term reasoning, natural language comprehension, and the available prior knowledge of LLMs for increased resilience and adaptation in autonomous mobile robots. We introduce REAL, an approach for REsilience and Adaptation using LLMs. REAL interfaces LLMs with the mission planning and control framework of an autonomous robot. The LLM employed by REAL provides (i) a source of prior knowledge to increase resilience for challenging scenarios that the system has not been explicitly designed for; (ii) a way to interpret natural language and other log/diagnostic information available in the autonomy stack, for mission planning; (iii) a way to adapt the control inputs using minimal user-provided prior knowledge about the robot. We integrate REAL in the autonomy stack of a real multirotor, querying onboard an offboard LLM at about $1.0-0.1 \mathrm{~Hz}$ as part of the robot’s mission planning and control feedback loops. We provide a demonstration of capabilities by showcasing in real-world experiments the ability of the LLM to reduce the position tracking errors of a multirotor, and decision-making to avoid potentially dangerous scenarios (e.g., robot oscillates) that are not explicitly accounted for in the initial prompt design.

Topics & Concepts