Inductive Coding with ChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories
Clarissa Sabrina Arlinghaus, Charlotte Wulff, Günter W. Maier
Abstract
Qualitative data is invaluable, yet its analysis is very time-consuming. To prevent the loss of valuable information and to streamline the coding process for developing and assigning inductive categories, we introduce LLM-Assisted Inductive Categorization (LAIC), a novel method of categorizing text responses using a Large Language Model (LLM). In two pre-registered studies, we tested two Generative Pre-trained Transformer (GPT) models that are commonly used in ChatGPT (GPT-3.5 Turbo and GPT-4o) across three temperature settings (0, 0.5, 1) with 10 repetitions each (120 runs in total). Outputs were evaluated based on established qualitative research criteria (credibility, dependability, confirmability, transferability, transparency). Two human coders also generated inductive categories and assigned text responses accordingly for comparison. Our findings demonstrate that both GPT models are highly effective in developing and assigning inductive categories, even outperforming human coders in agreement rates. Overall, GPT-4o achieved the best results (e.g., better explanations and higher agreement) and is recommended for inductive category formation and assignment with a temperature setting of 0 and 10 repetitions. This approach saves significant time and resources while enhancing analysis quality. Instructions and Python scripts for applying our new coding technique are freely available under a CC-BY 4.0 International license: https://osf.io/h4dux/