.png){kind=icon}
返回
Seformer: a long sequence time-series forecasting model based on binary position encoding and information transfer regularization
.png){kind=icon}
PRE
AI
DOI：10.1007/s10489-022-04263-z.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Long sequence time-series forecasting (LSTF) problems, such as weather forecasting, stock market forecasting, and power resource management, are widespread in the real world. The LSTF problem requires a model with high prediction accuracy. Recent studies have shown that the transformer model architecture is the most promising model structure for LSTF problems compared with other model architectures. The transformer model has the property of permutation equivalence, which leads to the importance of sequence position encoding, an essential process in model training. Currently, the continuous dynamics models constructed for position encoding using the neural differential equations (neural ODEs) method can model sequence position information well. However, we have found that there are some limitations when neural ODEs are applied to the LSTF problem, including the time cost problem, the baseline drift problem, and the information loss problem; thus, neural ODEs cannot be directly applied to the LSTF problem. To address this problem, we design a binary position encoding-based regularization model for long sequence time-series prediction, named Seformer, which has the following structure: 1) The binary position encoding mechanism, including intrablock and interblock position encoding. For intrablock position encoding, we design a simple ODE method by discretizing the continuum dynamics model, which reduces the time cost required to compute neural ODEs while maintaining their dynamics properties to the maximum extent. In interblock position encoding, a chunked recursive form is adopted to alleviate the baseline drift problem caused by eigenvalue explosion. 2) Information transfer regularization mechanism: By regularizing the model intermediate hidden variables as well as the encoder-decoder connection variables, we can reduce information loss during the model training process while ensuring the smoothness of the position information. Extensive experimental results obtained on six large-scale datasets show a consistent improvement in our approach over the baselines.
Keyword:
Long sequence time-series forecasting
Transformer
Position encoding
Regularization method
Conditional variational autoencoder
期刊
IF:
3.5
论文数:
7.6K
被引数:
1.7W
机构
引用论文
Forecasting COVID-19 outbreak progression using hybrid polynomial-Bayesian ridge regression model.png)
APPLIED INTELLIGENCE
OAAI
APPLIED INTELLIGENCEIF3.5
OAAI分享收藏A composite catalytic film of PEDOT:PSS/TiN–NPs on a flexible counter-electrode substrate for a dye-sensitized solar cell用于染料敏化太阳能电池的柔性对电极基底上的PEDOT:PSS/tin-nps复合催化膜
PREAI
分享收藏DSTP-RNN: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series predictionDstp-rnn: 用于长期和多变量时间序列预测的基于双阶段两阶段注意力的递归神经网络
PREAI
分享收藏Graph transformer network with temporal kernel attention for skeleton-based action recognition基于时间核关注的图变换网络在骨架动作识别中的应用OAAI
OAAI分享收藏
查看更多内容