Enhancing Attention Models via Multi-head Collaboration

Abstract

Neural attention based models have recently boosted performance on various NLP tasks. Compared with single-head attention, multi-head attention is more powerful and popular. Multi-head attention independently attend to information from different feature subspaces and generates multiple attention distributions. In this paper, we make an assumption that the current multi-head attention method can generate complementary attention distributions, but these distributions may not collaborate properly to improve prediction quality. To validate our assumption, we propose a simple but effective method to enhance the collaboration of different attention heads, which allows different heads to have the chance to rectify their attention scores with other heads. Empirical study shows that our proposed method can significantly improve the performance of multi-head attention over a range of NLP tasks, and the experimental results also prove the existence of the problem of multi-head collaboration.

Topics & Concepts