Usability and User eXperience Evaluation of Conversational Systems
Guilherme Corredato Guerino, Natasha Valentim
Abstract
Conversational Systems (CSs) are software that uses the user's voice to perform some action. Before these systems were available to users, they must be evaluated. In this sense, Usability and User eXperience (UX) evaluations contribute to the verification of software quality, since they evaluate several aspects such as efficiency, effectiveness, immersion, and user satisfaction. Therefore, the goal of our Systematic Mapping Study (SMS) is to identify the evaluation technologies (methods, techniques, models, among others) used by researchers and professionals to evaluate Usability and UX of CSs. We selected 39 papers for data extraction and, based on these works, we identified 31 different evaluation technologies. Besides, our SMS extracted the characteristics of technologies, CSs, and empirical studies described in the papers. Our results identify a lack of evaluation technologies of CSs that unite the concepts of Usability and UX and undergo empirical evaluations. Moreover, we observed researchers tend to create their questionnaires according to the needs of the study. Overall, our SMS presents data about the researched topic, describing the gaps, and contributing to the scientific community that evaluates Usability and UX of CSs.