DSFormer-LRTC: Dynamic Spatial Transformer for Traffic Forecasting With Low-Rank Tensor Compression

Abstract

Traffic flow forecasting is challenging due to the intricate spatio-temporal correlations in traffic patterns. Previous works captured spatial dependencies based on graph neural networks and used fixed graph construction methods to characterize spatial relationships, which limits the ability of models to capture dynamic and long-range spatial dependencies. Meanwhile, prior studies did not consider the issue of a large number of redundant parameters in traffic prediction models, which not only increases the storage cost of the model but also reduces its generalization ability. To address the above challenges, we propose a Dynamic Spatial Transformer for Traffic Forecasting with Low-Rank Tensor Compression (DSFormer-LRTC). Specifically, we constructed a global spatial Transformer to capture remote spatial dependencies, and a distance-based mask matrix is used in local spatial Transformer to enhance the adjacent spatial influence. To reduce the complexity of the model, the model adopts a design that separates temporal and spatial. Meanwhile, we introduce low-rank tensor decomposition to reconstruct the parameter matrix in Transformer module to compress the proposed model. Experimental results show that DSFormer-LRTC achieves state-of-the-art performance on four real-world datasets. The experimental analysis of attention matrix also proves that the model can learn dynamic and distant spatial features. Finally, the compressed model parameters reduce the original parameter size by two-thirds, while significantly outperforming the baseline model in terms of computational efficiency.

Topics & Concepts