LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
Qi Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang
Abstract
Long-Context Question Answering (LCQA), a challenging task, aims to reason over longcontext documents to yield accurate answers to questions.Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue.Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence.However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise.To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details).We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs.Extensive experiments on three multihop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%).Furthermore, we conduct quantitative ablation studies and multidimensional analyses, highlighting the effectiveness of the system's components and finetuning strategies.Data and code are available at https://github.com/QingFei1/LongRAG.