Gaussian-based Interval-Aware Transformer With Interval Embedding for Data Sequence Modeling With Irregular Sampling Frequency in Industrial Processes
Ziyi Yang, Kai Wang, Xiaofeng Yuan, Yalin Wang, Chunhua Yang, Weihua Gui, Lingjian Ye, Feifan Shen
Abstract
Temporal feature representation is critical for soft sensor modeling in industrial time sequences. Deep learning networks like long short-term memory are often used to model the temporal dynamics of data sequences. However, the data collected from industrial plants are usually sampled with irregular frequency, making it challenging for traditional methods to handle these temporally changeable relationships. Therefore, a Gaussian-based interval-aware transformer (GIA-Trans) with interval embedding is proposed in this article to model industrial data with irregular sampling frequency. In GIA-Trans, positional and temporal embedding layers are established to take positional distances and time intervals of samples into account. Then, Gaussian-based time-aware attention is proposed to tackle the changeable time intervals with adaptive weights. In this way, the temporal correlations between samples can be adaptively captured. The GIA-Trans is applied to an industrial hydrocracking process to predict the C5 and C6 content of light naphtha.