Dual Cross-Attention Transformer Networks for Temporal Predictive Modeling of Industrial Process

Abstract

Industrial Predictive Modeling plays an important role in process control and optimization. Industrial process data arisen in the real-world applications often involves nonlinear and temporal characters, which are two main challenges for accurate industrial predictive modeling. While the previous transformer-based industrial predictive models only consider the temporal information of the industrial time series data, however, the different importance of the process variables is generally ignored. In this paper, we propose a novel Dual Cross-Attention-based Transformer (DCAFormer) to capture both the cross-time dependencies and the cross-variable dependencies in parallel for better predictability. Specifically, the proposed DCAFormer is composed of cross-time self-attention layer and cross-variable self-attention layer. The cross-variable self-attention is developed to capture multivariate correlations by inverting the input time series into variate tokens. The De-stationary cross-time self-attention is employed to extract the intrinsic non-stationary information into the temporal dependencies from the time series data. The comparative as well as the ablation experiments are conducted on the real-world aluminum electrolysis process. The experimental results show that DCAFormer achieve better prediction performance than other competitive transformer models.

Topics & Concepts