Next Generation XR Systems—Large Language Models Meet Augmented and Virtual Reality
Muhammad Zeshan Afzal, Sk Aziz Ali, Didier Stricker, Peter Eisert, Anna Hilsmann, Daniel Pérez-Marcos, Marco Bianchi, Sonia Crottaz‐Herbette, Roberto De Ioris, Eleni Mangina, Mirco Sanguineti, Ander Salaberria, Oier López de Lacalle, Aitor García Pablos, Montse Cuadros
Abstract
Extended reality (XR) is evolving rapidly, offering new paradigms for human-computer interaction. This position paper argues that integrating large language models (LLMs) with XR systems represents a fundamental shift toward more intelligent, context-aware, and adaptive mixed-reality experiences. We propose a structured framework built on three key pillars: first, perception and situational awareness, second, knowledge modeling and reasoning, and third, visualization and interaction. We believe leveraging LLMs within XR environments enables enhanced situational awareness, real-time knowledge retrieval, and dynamic user interaction, surpassing traditional XR capabilities. We highlight the potential of this integration in neurorehabilitation, safety training, and architectural design while underscoring ethical considerations, such as privacy, transparency, and inclusivity. This vision aims to spark discussion and drive research toward more intelligent, human-centric XR systems.