UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems
Jafar Afzali, Aleksander Mark Drzewiecki, Krisztian Balog, Shuo Zhang
Abstract
We present an extensible user simulation toolkit to facilitate automatic evaluation of conversational recommender systems. It builds on an established agenda-based approach and extends it with several novel elements, including user satisfaction prediction, persona and context modeling, and conditional natural language generation. We showcase the toolkit with a pre-existing movie recommender system and demonstrate its ability to simulate dialogues that mimic real conversations, while requiring only a handful of manually annotated dialogues as training data.
Topics & Concepts
Computer scienceRecommender systemPersonaContext (archaeology)User satisfactionHuman–computer interactionWorld Wide WebDialog systemArtificial intelligenceDialog boxBiologyPaleontologySpeech and dialogue systemsTopic ModelingRecommender Systems and Techniques