让注意力机制动态适应多变量时间序列中的复杂关系，提升预测精度。
Dynamic Relational Priming Improves Transformer in Multivariate Time Series
- 通过可学习调制动态调整每对令牌的表示，捕捉不同通道间的独特关系。
- 在多个基准上最高提升6.5%预测准确率，且序列长度可减少40%。
- 适合处理物理规律或时序动态差异大的多变量时间序列任务。
Transformer 中的标准注意力机制使用静态的令牌表示，在每一层中所有成对计算中保持不变，这限制了其与可能多样化的令牌对关系动态的匹配能力。尽管在关系相对同质的领域表现优异，但标准注意力在多变量时间序列（MTS）数据中难以捕捉不同通道对之间异质的依赖关系——同一系统内的不同通道对可能遵循完全不同的物理定律或时序动态。为此，我们提出带有动态关系引导（prime attention）的注意力机制。不同于标准注意力中每个令牌在所有成对交互中呈现相同表示，prime attention 通过可学习调制动态调整每个令牌（按交互），以最好地捕捉每对令牌的独特关系动态，从而优化特定关系的成对交互。这种表示上的可塑性使 prime attention 能有效提取 MTS 中的关系特异性信息，同时保持与标准注意力相同的渐近计算复杂度。实验结果表明，prime attention 在多个基准上持续优于标准注意力，预测准确率最高提升 6.5%。此外，prime attention 在使用最多 40% 更短序列长度的情况下，仍能达到相当或更优的性能，进一步证明其优越的关系建模能力。
原文摘要 · Abstract (English)
Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their representational alignment with the potentially diverse relational dynamics of each token-pair interaction. While they excel in domains with relatively homogeneous relationships, standard attention's static relational learning struggles to capture the diverse, heterogeneous inter-channel dependencies of multivariate time series (MTS) data--where different channel-pair interactions within a single system may be governed by entirely different physical laws or temporal dynamics. To better align the attention mechanism for such domain phenomena, we propose attention with dynamic relational priming (prime attention). Unlike standard attention where each token presents an identical representation across all of its pair-wise interactions, prime attention tailors each token dynamically (or per interaction) through learnable modulations to best capture the unique relational dynamics of each token pair, optimizing each pair-wise interaction for that specific relationship. This representational plasticity of prime attention enables effective extraction of relationship-specific information in MTS while maintaining the same asymptotic computational complexity as standard attention. Our results demonstrate that prime attention consistently outperforms standard attention across benchmarks, achieving up to 6.5\% improvement in forecasting accuracy. In addition, we find that prime attention achieves comparable or superior performance using up to 40\% less sequence length compared to standard attention, further demonstrating its superior relational modeling capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。