AI-based novelty detection in crowdsourced idea spaces
Julian Just, Thomas Ströhle, Johann Füller, Katja Hutter
Abstract
Processing large and heterogeneous numbers of ideas submitted to crowdsourcing contests is a regular challenge for idea evaluators. The aim of this study is to investigate a potential use case for AI-based innovation management and to extend the knowledge of using automated novelty detection in idea evaluation processes. AI-based language models can automatically allocate short texts according to their semantic similarity in an embedded space. We represent the semantic content of crowdsourced ideas with the three contemporary text embeddings – Doc2Vec, SBERT, and GPT-3-based Ada Similarity – and compute their semantic distance to different reference sets using different novelty detection algorithms. We then compare the algorithm-generated scores with human novelty assessments to validate them. While selected novelty scores based on text embeddings correlate with humans, our results show that scores based on SBERT embeddings best match human novelty assessments. We also find that AI-based novelty detection approaches perform better for ideas below the median word count and when compared to a set of existing solutions, suggesting that the chosen language model is not the only factor influencing the applicability of the proposed approach. Furthermore, the study highlights important features and limitations of automatically generated novelty scores that need to be considered when complementing evaluators searching for new ideas in crowdsourcing contests and beyond.