Optimizing Attention in a Transformer for Multihorizon, Multienergy Load Forecasting in Integrated Energy Systems
Jili Fan, Wei Zhuang, Min Xia, Wenxuan Fang, Jun Liu
Abstract
Accurate forecasting of multienergy loads is essential for designing, operating, scheduling, and managing integrated energy systems (IESs). Recent research suggests that transformer models have the potential to improve long-sequence predictions. However, existing transformer models often emphasize capturing temporal dependencies while neglecting crucial dependencies among different variables necessary for multienergy load forecasting. Moreover, transformer models encounter challenges related to quadratic time complexity and significant memory usage, which hinder their direct applicability to tasks involving long-sequence, multienergy load forecasting. To tackle these challenges, we propose a model called DTformer and apply it to the task of multihorizon, multienergy load forecasting in IES. Within DTformer, we employ patch embedding to convert the input multienergy load sequences into a 3-D vector array, preserving both temporal and variable information. Subsequently, we propose the temporal top windowed attention (TWA) module and the dual variable attention module to handle extended temporal dependencies and intervariable dependencies. Importantly, the computational complexity and memory requirements of the TWA model are regulated at a level of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(N^\frac{4}{3})$</tex-math></inline-formula> . Through extensive experimentation, we found that our DTformer surpasses baseline models in terms of performance using the IES dataset sourced from Arizona State University's Tempe campus.