Hybrid Big Data-LLM Framework for Intelligent Scientific Literature Mining
Ishtiaque Alam, Ruhul Amin Md Rashed, Partha Chakraborty, Sraboni Clara Mohonta, Hasan Imam, Mohammad Muzahidur Rahman Bhuiyan
Abstract
The rapid increase in scientific publications has posed a serious problem in the effective discovery, analysis and synthesis of pertinent knowledge under various fields of research. The conventional methods of literature mining usually find it difficult to handle large heterogeneous datasets preserving textual background and semantic precision. In order to overcome these shortcomings, this paper will present Hybrid Big Data-Large Language Model (LLM) Framework to Intelligent Scientific Literature Mining that combines scalable distributed data processing with state-of-the-art transformer-based language intelligence. The suggested framework uses big data technologies to ingest, store, and process scholarly repositories in high throughput, and has modules powered by LLMs to understand the semantic information in deep, and to summarize and analyse citations, topic modelling, and research trends. The multi-layer architecture that incorporates data orchestration, neural knowledge representation, and adaptive retrieval mechanisms is presented to improve the accuracy and minimize the information overload. Moreover, the framework supports embedding strategy and reinforcement driven filtering to enhance ranking of relevance and reduce redundancy in the extracted insights. The experimental analysis shows that the hybrid system has a much higher semantic accuracy, retrieval efficiency, scalability and rate of knowledge discovery compared to the traditional literature mining systems. Findings have shown enhanced contextual mapping of research subjects and easy discovery of new scientific pathways. The framework is also useful in facilitating interdisciplinary exploration in the sense that it allows automatic connectivity between knowledge spaces that used to be unconnected. The study will provide an intelligent and scalable scientific discovery paradigm of next-generation scientific discovery systems, which enable accelerated innovation processes and evidence-based research. The model is especially applicable when dealing with data-intensive environments, in which it is important that researchers, institutions and policy-makers quickly gain an understanding of the changing literature.