Analyzing the Text Contents Produced by ChatGPT: Prompts, Feature-Components in Responses, and a Predictive Model
Leping Liu
Abstract
ChatGPT is a large language model that uses deep learning to produce natural language and generate intelligent and relevant responses to user prompts. It comes to the field of education as an inevitable wave. Educators have to deal with it and figure out appropriate ways to use it and produce positive learning. This study explores the use of ChatGPT from the perspective of front-end users, focusing on the text-content that ChatGPT can produce for learners to learn new knowledge (e.g., a concept, a theory, or an application). The sample of this study consists of 253 ChatGPT text responses derived from three types of initial prompts/questions: general questions, specific questions, and questions with interactive prompts. Six feature components of text-information that can help learners to understand new knowledge are analyzed (concept and definition, procedure, example, comparison or contrast, deductive or inductive argument, summary). The results from Chi-square tests indicate that the presence of each feature component in the responses differs by the types of prompts. The results from a logistic regression analysis reveal that the presence of five (out of the six) feature components are significant to the probability that a response providing accurate and reliable information. The integration of using ChatGPT into learning is discussed. Further research questions are suggested.