Gradient-based Analysis of NLP Models is Manipulable
Junlin Wang, Jens Tuyls, Eric Wallace, Sameer Singh
Abstract
Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness. In this paper, however, we demonstrate that the gradients of a model are easily manipulable, and thus bring into question the reliability of gradient-based analyses.
Topics & Concepts
Computer scienceMerge (version control)Artificial intelligenceAdversarial systemSimplicityNatural language processingFlexibility (engineering)Pattern recognition (psychology)Machine learningInformation retrievalMathematicsStatisticsEpistemologyPhilosophyAdversarial Robustness in Machine LearningExplainable Artificial Intelligence (XAI)Topic Modeling