LumberChunker: Long-Form Narrative Document Segmentation
André V. Duarte, João Marques, Miguel Graça, Miguel Freire, Lei Li, Arlindo L. Oliveira
Abstract
Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information.We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content's semantic independence is better captured.We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift.To evaluate our method, we introduce GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg 1 .Our experiments show that Lum-berChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro.