Optimizing Transformer Models for Resource-Constrained Environments: A Study on Model Compression Techniques
Ziqian Luo, Hanrui Yan, Xueting Pan
Abstract
Recent progress in computer vision has been driven by transformer-based models, which consistently outperform traditional methods across various tasks. However, their high computational and memory demands limit their use in resource-constrained environments. This research addresses these challenges by investigating four key model compression techniques: quantization, low-rank approximation, knowledge distillation, and pruning. We thoroughly evaluate the effects of these techniques, both individually and in combination, on optimizing transformers for resource-limited settings. Our experimental findings show that these methods can successfully strike a balance between accuracy and efficiency, enhancing the feasibility of transformer models for edge computing.