ECHO: A Tool for Empirical Evaluation Cloud Chatbots
Abdur Rahim Mohammad Forkan, Prem Prakash Jayaraman, Yong‐Bin Kang, Ahsan Morshed
Abstract
A chatbot is a software that interacts with humans by conducting conversations via textual or auditory methods. Chatbots have recently been used for plethora of applications including travel, medical, education, retail etc. Several cloud-based platforms (e.g. IBM, Amazon, Google, Microsoft) are available for developing and deploying chatbots. However, there is a lack of an evaluation methodology and a tool for evaluating chatbots comprehensively. Current approaches for comparing cloud-based chatbots are manual and rely on expert's judgement. In this short paper, we propose, devise, implement and demonstrate a tool namely ECHO for empirical evaluation of cloud-based chatbots. ECHO is capable of automatically evaluating multiple cloud-based chatbots and report the outcomes of the comparative evaluation. We validate the efficacy of ECHO by conducting comparative evaluation of 3 popular cloud-based chatbots in 2 different question-answering application scenarios with 3 levels of complexities.