给Transformer加了个时间偏移权重，让模型更关注近期数据。
Recency Biased Causal Attention for Time-series Forecasting
- 用重尾衰减函数重加权注意力，引入时间最近优先的偏好。
- 在多个时间序列预测基准上表现优于标准Transformer。
- 适合需要捕捉短期动态的时序建模任务，如金融、气象预测。
时间序列建模中，近期信息通常更重要，这种‘近期偏向’是有效的归纳先验：它强调邻近观测值，同时仍能保留长程依赖。标准Transformer注意力依赖全连接交互，忽视了时序数据的因果性和局部结构。本文提出一种简单机制，通过平滑重尾衰减对注意力分数进行重加权，增强局部时间依赖性，同时不牺牲捕捉更广范围和数据特定相关性的灵活性。实验表明，引入近期偏向的注意力可一致提升序列建模性能，使Transformer更接近RNN的‘读取、忽略、写入’操作模式。最终，该方法在多个具有挑战性的时间序列预测基准上取得竞争力甚至更优的表现。
原文摘要 · Abstract (English)
Recency bias is a useful inductive prior for sequential modeling: it emphasizes nearby observations and can still allow longer-range dependencies. Standard Transformer attention lacks this property, relying on all-to-all interactions that overlook the causal and often local structure of temporal data. We propose a simple mechanism to introduce recency bias by reweighting attention scores with a smooth heavy-tailed decay. This adjustment strengthens local temporal dependencies without sacrificing the flexibility to capture broader and data-specific correlations. We show that recency-biased attention consistently improves sequential modeling, aligning Transformer more closely with the read, ignore, and write operations of RNNs. Finally, we demonstrate that our approach achieves competitive and often superior performance on challenging time-series forecasting benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。