Socratic Video Understanding on Unmanned Aerial Vehicles
I. de Zarzà, J. de Curtò, Carlos T. Calafate
Abstract
In this work, we propose a system for video understanding through zero-shot reading comprehension using Socratic Models. Specifically, we create a language-based world-state history of events and objects present in a scene captured by an Unmanned Aerial Vehicle (UAV). To achieve this, video footage from RYZE Tello microdrones is transmitted to a ground computer for further processing. The semantically rich information offered by Large Language Models (LLMs) enables open-ended reasoning, such as event forecasting with minimal human intervention, in a cost-effective robotic system. BLIP-2 is employed to answer a given set of instructional prompts, creating a log-state of objects, humans, and hazards that can be searched. Simultaneously, it suggests probable actions in the scene and can assist the human controller with an estimated best command. The BLIP-2 instructional prompts are then combined with OpenAI's da-vinci-003/gpt-3.5-turbo to generate comprehensive video descriptions and summarize likely actions. The LLM-enhanced generated texts achieve a GUNNING Fog median grade level in the range of 7-12.