Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort
2021Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing76 citationsDOIOpen Access PDF
Abstract
Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges -namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token.
Topics & Concepts
Quantization (signal processing)Computer scienceTransformerComputer engineeringResidualAlgorithmComputer hardwareElectrical engineeringEngineeringVoltageTopic ModelingMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning