用非线性相关性改进时间序列注意力，提升预测精度
XicorAttention: Time Series Transformer Using Attention with Nonlinear Correlation
- 用查特吉等级相关系数替代传统注意力中的矩阵乘法
- 在真实数据集上预测误差降低最高达9.1%
- 适合需要捕捉复杂非线性依赖的时间序列任务
针对时间序列预测中现有Transformer模型对非线性依赖建模不足的问题，本文提出一种基于查特吉等级相关系数的新型注意力机制XicorAttention。该方法将标准注意力中的矩阵乘法替换为能衡量变量间非线性关系的等级相关系数，并引入可微近似SoftSort和SoftRank以实现端到端训练。将XicorAttention集成到多个前沿Transformer模型中，在多个真实世界数据集上的实验表明，引入非线性相关性可使预测精度提升最高约9.1%。
原文摘要 · Abstract (English)
Various Transformer-based models have been proposed for time series forecasting. These models leverage the self-attention mechanism to capture long-term temporal or variate dependencies in sequences. Existing methods can be divided into two approaches: (1) reducing computational cost of attention by making the calculations sparse, and (2) reshaping the input data to aggregate temporal features. However, existing attention mechanisms may not adequately capture inherent nonlinear dependencies present in time series data, leaving room for improvement. In this study, we propose a novel attention mechanism based on Chatterjee's rank correlation coefficient, which measures nonlinear dependencies between variables. Specifically, we replace the matrix multiplication in standard attention mechanisms with this rank coefficient to measure the query-key relationship. Since computing Chatterjee's correlation coefficient involves sorting and ranking operations, we introduce a differentiable approximation employing SoftSort and SoftRank. Our proposed mechanism, ``XicorAttention,'' integrates it into several state-of-the-art Transformer models. Experimental results on real-world datasets demonstrate that incorporating nonlinear correlation into the attention improves forecasting accuracy by up to approximately 9.1\% compared to existing models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。