A Comparative Analysis of Large Language Models with Retrieval-Augmented Generation based Question Answering System
Hetul Niteshbhai Patel, Azara Surti, Parth Goel, Bankim Patel
Abstract
In recent studies, Large Language Models (LLMs) have shown remarkable effectiveness in a wide range of natural language processing tasks. However, their knowledge is limited to the data they were trained on, which may not cover context-aware information. This limitation reduces their capability to provide precise and specific information apart from other parameters, especially in specialized fields. Retrieval Augmented Generation (RAG) Systems utilizes to overcome this limitation by enhancing the capabilities of LLMs through the retrieval of relevant information from external sources during the generation process. This research work presents the comparative analysis of the performance of three popular LLMs’ – GPT-3.5-turbo from OpenAI, Gemini-Pro from Google, LLama3 by Meta when integrated into RAG system for question answering application. The study contrasts the efficiency of these LLM in generating relevant response with the aid of retrieved information in Q&A task. Along with these three LLMs other embedding models such as Text-embedding-ada-002-v2, embedding-001 and nomic-embed-text embedding model are used. Seven evaluation matrices are used from the RAGAS evaluation framework on self-created dataset to assess the performance of the RAG QA systems. The result showed that the gpt-3.5-turbo Large Language model achieved highest performance outperforming Gemini Pro by 5.66% and Llama3 by 8.40%.