通过逻辑启发正则化，让时间序列模型只关注关键依赖关系。
Enhancing Time Series Forecasting via Logic-Inspired Regularization
- 从逻辑角度定义有效依赖，引导模型聚焦关键时间点。
- 提出注意力稀疏正则化方法，使注意力图更简洁高效。
- 无需修改结构，可直接提升主流Transformer模型性能。
时间序列预测在诸多应用中至关重要。基于Transformer的方法是主流技术，但现有方法对所有标记依赖一视同仁。我们发现，不同预测场景下标记依赖的有效性存在差异，而现有方法忽略了这一区别，影响了性能。这引发两个问题：(1) 什么是有效的标记依赖？(2) 如何学习这些有效依赖？从逻辑视角出发，我们将基于Transformer的时间序列预测方法与逻辑框架对齐，将有效依赖定义为能保证标记作为原子公式的依赖（问题1）。随后，将Transformer的学习过程与逻辑中获取原子公式的过程对齐，启发我们设计一种学习有效依赖的方法（问题2）。具体地，我们提出注意力逻辑正则化（Attn-L-Reg），一种即插即用的方法，通过使注意力图稀疏，引导模型使用更少但更有效的依赖，从而确保标记作为原子公式，提升预测性能。大量实验与理论分析验证了Attn-L-Reg的有效性。
原文摘要 · Abstract (English)
Time series forecasting (TSF) plays a crucial role in many applications. Transformer-based methods are one of the mainstream techniques for TSF. Existing methods treat all token dependencies equally. However, we find that the effectiveness of token dependencies varies across different forecasting scenarios, and existing methods ignore these differences, which affects their performance. This raises two issues: (1) What are effective token dependencies? (2) How can we learn effective dependencies? From a logical perspective, we align Transformer-based TSF methods with the logical framework and define effective token dependencies as those that ensure the tokens as atomic formulas (Issue 1). We then align the learning process of Transformer methods with the process of obtaining atomic formulas in logic, which inspires us to design a method for learning these effective dependencies (Issue 2). Specifically, we propose Attention Logic Regularization (Attn-L-Reg), a plug-and-play method that guides the model to use fewer but more effective dependencies by making the attention map sparse, thereby ensuring the tokens as atomic formulas and improving prediction performance. Extensive experiments and theoretical analysis confirm the effectiveness of Attn-L-Reg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。