What did the occupant say? Fine-tuning and evaluating a large language model for efficient analysis of multi-domain indoor environmental quality feedback

Abstract

• Fine-tuned a large language model for classifying IEQ occupant feedback. • The model was developed using 14,622 manually labelled occupant feedback. • The model achieved a prediction accuracy of 93 % based on the evaluation data. • The model minimises the risk of false positive and false negative predictions of IEQ feedback. • The model effectively distinguishes feedback using similar words in IEQ and non-IEQ context. Qualitative feedback from occupants on indoor environmental quality (IEQ) in unstructured text can provide valuable insights into the causes of comfort and discomfort in buildings. This feedback can be collected from open-ended survey questions, interviews, crowdsourced data, or innovative home automation technology that can transform voice inputs into text. However, manual text data processing is time-consuming and requires significant efforts to extract relevant insights, such as text classification into IEQ categories (i.e., visual, thermal, air quality and acoustic). Most IEQ studies that automated text feedback classification into IEQ categories relied on keyword matching, which cannot understand the context of some keywords, potentially leading to incorrect classification. To address this issue, we automated the detection and categorisation of unstructured IEQ feedback by adopting the Bidirectional Encoder Representations from Transformers (BERT) language model architecture and fine-tuning it on 14,622 manually labelled IEQ text feedback. The resulting model, IEQ-BERT, achieved a prediction accuracy of 93 % and macro average precision, recall, and F1-scores of 0.93, 0.94, and 0.93, respectively, across the five considered classes (i.e., acoustic, indoor air quality, thermal, visual, or No IEQ). Therefore, the model can effectively distinguish text concerning IEQ and identify which IEQ domain – acoustic, indoor air quality, thermal, visual, or their combinations – is being reported. IEQ-BERT can be used alone or integrated into building automation systems to identify patterns and trends of occupant feedback, prioritise areas for improvement, and support the development of targeted strategies to improve IEQ. This research contributes to developing efficient methods for analysing occupant feedback, ultimately leading to improved building performance and occupant quality of life.

Topics & Concepts