Spectrum-BERT: Pretraining of Deep Bidirectional Transformers for Spectral Classification of Chinese Liquors
Yansong Wang, Yundong Sun, Yansheng Fu, Dongjie Zhu, Zhaoshuo Tian
Abstract
Counterfeit Chinese liquor incidents in China have significantly disrupted market order and jeopardized the health of consumers. Currently, deep learning-based spectral detection techniques are extensively leveraged in non-invasive food inspection. Excessive reliance on labels severely limits its application in real scenarios. To make better use of limited samples, we are the first to use the “unsupervised pre-training & supervised fine-tuning” paradigm in combining the Transformer architecture for feature extraction and classification of the Chinese liquor spectrum, and propose Spectrum-BERT, which represents Bidirectional Encoder Representations from Transformers for Spectrum. Specifically, we creatively propose spectral curve partitioning and 1-D convolutional layer mapping to maintain the model’s sensitivity to characteristic peak locations and local information of spectral curves. Moreover, the paradigm of “unsupervised pre-training & supervised fine-tuning” addresses the limitation of label deficiency, thus improving the model’s applicability. Finally, we have conducted extensive experiments on the real liquor spectral dataset. Comparative experiments demonstrate that Spectrum-BERT outperforms all baselines on all metrics using only 70% supervised signal. The limit experimental results show that Spectrum-BERT can still maintain its lead using only the 10% supervised signal. Thanks to the more efficient model architecture, Spectrum-BERT’s model parameters and FLOPs are only 1/568 and 1/322 of those of baselines.