How the Choice of LLM and Prompt Engineering Affects Chatbot Effectiveness
Łukasz Pawlik
Abstract
Modern businesses increasingly rely on chatbots to enhance customer communication and automate routine tasks. The research aimed to determine the optimal configurations of a telecommunications chatbot on the Rasa Pro platform, including the selection of large language models (LLMs), prompt formats, and command structures. The impact of various LLMs, prompt formats, and command precision on response quality was analyzed. Smaller models, like Gemini-1.5-Flash-8B and Gemma2-9B-IT, can achieve results comparable to larger models, offering a cost-effective solution. Specifically, the Gemini-1.5-Flash-8B model achieved an accuracy improvement of 21.62 points when using the JSON prompt format. This emphasizes the importance of prompt engineering techniques, like using structured formats (YAML, JSON) and precise commands. The study utilized a dataset of 400 sample test phrases created based on real customer service conversations with a mobile phone operator’s customers. Results suggest optimizing chatbot performance does not always require the most powerful models. Proper prompt preparation and data format choice are crucial. The theoretical framework focuses on the interaction between model size, prompt format, and command precision. Findings provide insights for chatbot designers to optimize performance through LLM selection and prompt construction. These findings have practical implications for businesses seeking cost-effective and efficient chatbot solutions.