MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, Tiejun Huang
Abstract
Processing long contexts presents a significant challenge for large language models (LLMs). While recent advancements allow LLMs to handle much longer contexts than before (e.g., 32K or 128K tokens), it is computationally expensive and can still be insufficient for many applications. Retrieval-Augmented Generation (RAG) is considered a promising strategy to address this problem. However, conventional RAG methods face inherent limitations because of two underlying requirements: 1) explicitly stated queries, and 2) well-structured knowledge. These conditions, however, do not hold in general long-context processing tasks.
Topics & Concepts
Boosting (machine learning)Computer scienceContext (archaeology)Computer architectureArtificial intelligenceBiologyPaleontologyAdvanced Image and Video Retrieval TechniquesMultimodal Machine Learning ApplicationsImage Retrieval and Classification Techniques