Enabling GPTs for Expert-Level Environmental Engineering Question Answering
Jun‐Jie Zhu, Meiqi Yang, Jinyue Jiang, Yiming Bai, Danqi Chen, Zhiyong Jason Ren
Abstract
Artificial intelligence (AI) holds significant potential for advancing research and development in the field of environmental science and engineering (ESE), but the development of domain-specific large language models (LLMs) in this field has not been reported. This study addresses this gap by evaluating the performance of advanced LLMs in answering expert-level, closed-book environmental engineering questions. We assessed two generative pretrained transformer (GPT) models and five fine-tuned models (FTMs) on an expert-level question answering (QA) data set, focusing on relevance (from 0 to 1), factuality (0 to 1), format, richness, QA difficulty level, and domain topic. Results show that GPT-4 achieves a relevance score of 0.644 and a factuality score of 0.791 based on 286 questions, indicating room for improvement, particularly for more difficult questions (scores dropped to below 0.5). Notably, FTMs with larger data sets resisted factuality degradation, highlighting the need for high-quality training materials. Inaccuracies and format issues are often linked to overtraining and catastrophic interference. This first investigation leverages expert-level textbooks to enhance LLM performance, thereby providing valuable insights and setting the stage for developing more robust domain-specific LLMs for environmental applications.