Natural Language Processing-Driven Document Summarization Using Attention-Guided Memory-Augmented Transformer
Suresh Kurapati, K. Naveen Kumar, Rehaam Abdohwr, Subhojit Ghosh, Veeramachaneni Jhansi Lekha
Abstract
In recent years, Document summarization has become an important area in Natural Language Processing (NLP) with the growth of long-rich texts, such as the Government Report (GovReport). Traditional transformer-based models achieve strong results on short texts, although they face challenges with very long documents owing to the loss of contextual information; thus, the segmentation methods and chain-of-thought prompting improved coverage. Hence, this research proposes an Attention-Guided Memory-Augmented Transformer (AGMGT) for long-document summarization with the help of the GovReport dataset, which contains government research reports and their expert-written summaries. The documents are then segmented and encoded using transformer encoders to manage the context length. Furthermore, a memory bank is constructed by storing salient information identified through attention scoring. Subsequently, cross-memory attention is applied for chunk-level micro-summarization to maintain consistency across sections. Then, the micro-summaries are aggregated into a global outline to generate a final summary using hierarchical attention over the outlines. Finally, the model is trained with cross-entropy and auxiliary coverage losses to generate concise and factually correct summaries of long government reports. The proposed AGMGT achieved better results in terms of ROUGE-1 (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{0. 8 2 2 1}$</tex>), ROUGE-2 (0.6432), and ROUGE-L (0.7932) than the existing multilingual transformer 5 (mT5-Large) model.