Deceptive XAI: Typology, Creation and Detection
Johannes Schneider, Christian Meske, Michail Vlachos
Abstract
Abstract Providing rationales for decisions can enhance transparency and cultivate trust. Nevertheless, in light of economic incentives and other factors that may encourage manipulation, the reliability of such explanations comes into question. This manuscript builds upon a previous conference paper $$^*$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mrow/> <mml:mo>∗</mml:mo> </mml:msup> </mml:math> by introducing a conceptual framework for deceptive explanations and constructing a typology grounded in interdisciplinary literature. The focus of our work is on how AI models can generate and detect deceptive explanations. In our empirical evaluation, we focus on text classification and introduce modifications to the explanations generated by GradCAM, a well-established method for explaining neural networks. Through a user study comprising 200 participants, we demonstrate that these deceptive explanations have the potential to mislead individuals. However, we also demonstrate that machine learning (ML) techniques can discern even subtle deceptive tactics with an accuracy exceeding 80%, given sufficient domain expertise. Furthermore, even in the absence of domain knowledge, unsupervised learning can be employed to identify inconsistencies in the explanations, provided that fundamental information about the underlying predictive model is accessible.