A Difference Metric Attention With Position Distance-Based Weighting for Transformer in Data Sequence Modeling of Industrial Processes
Ziyi Yang, Kai Wang, Lingjian Ye, Xiaofeng Yuan, Yalin Wang, Chunhua Yang, Weihua Gui
Abstract
Accurate feature extraction and quality variable prediction are critical problems for time sequences in industrial processes. However, industrial samples often exhibit strong temporal correlations with each other that have different positional distances, making it challenging for conventional data-driven models like long short-term memory (LSTM) and Vanilla transformer to capture these underlying features. In this article, a difference metric attention with position distance-based weighting is proposed for transformer (DMA-trans) in industrial time series modeling. First, the DMA is established to calculate the difference of query-key vector pair in transformer to measure the spatial similarity. In this fashion, the difference can accurately represent the spatial similarity of vectors, compared with the original dot product directly on two vectors. Then, positional distance-based weights are designed to capture the sample relevance that has different positional distances. This may help to extract more potential features because the closer samples tend to have higher relevance while there may be weak correlations if two samples are far in positional distance. The effectiveness of the DMA-trans model is validated in industrial hydrocracking processes for C5 content of the light naphtha and the final boiling point of the jet fuel.