The Effectiveness of a Custom AI Chatbot for Type 2 Diabetes Mellitus Health Literacy: Development and Evaluation Study
Anthony Kelly, Eoin Noctor, Laura Ryan, Pepijn Van de Ven
Abstract
BACKGROUND: People living with chronic diseases are increasingly seeking health information online. For individuals with diabetes, traditional educational materials often lack reliability and fail to engage or empower them effectively. Innovative approaches such as retrieval-augmented generation (RAG) powered by large language models have the potential to enhance health literacy by delivering interactive, medically accurate, and user-focused resources based on trusted sources. OBJECTIVE: This study aimed to evaluate the effectiveness of a custom RAG-based artificial intelligence chatbot designed to improve health literacy on type 2 diabetes mellitus (T2DM) by sourcing information from validated reference documents and attributing sources. METHODS: A T2DM chatbot was developed using a fixed prompt and reference documents. Two evaluations were performed: (1) a curated set of 44 questions assessed by specialists for appropriateness (appropriate, partly appropriate, or inappropriate) and source attribution (matched, partly matched, unmatched, or general knowledge) and (2) a simulated consultation of 16 queries reflecting a typical patient's concerns. RESULTS: Of the 44 evaluated questions, 32 (73%) responses cited reference documents, and 12 (27%) were attributed to general knowledge. Among the 32 sourced responses, 30 (94%) were deemed fully appropriate, with the remaining 2 (6%) being deemed partly appropriate. Of the 12 general knowledge responses, 1 (8%) was inappropriate. In the 16-question simulated consultation, all responses (100%) were fully appropriate and sourced from the reference documents. CONCLUSIONS: A RAG-based large language model chatbot can deliver contextually appropriate, empathetic, and clinically credible responses to T2DM queries. By consistently citing trusted sources and notifying users when relying on general knowledge, this approach enhances transparency and trust. The findings have relevance for health educators, highlighting that patient-centric reference documents-structured to address frequent patient questions-are particularly effective. Moreover, instances in which the chatbot signals that it has drawn on general knowledge can provide opportunities for health educators to refine and expand their materials, ensuring that more future queries are answered from trusted sources. The findings suggest that such chatbots may support patient education, promote self-management, and be readily adapted to other health contexts.