对比固定与学习型时间编码在时序Transformer中的表现
Comparing Prior and Learned Time Representations in Transformer Models of Timeseries
- 比较固定时间编码与数据自学习时间编码的差异
- 固定编码在周期性任务中仍难有效捕捉先验知识
- 强调人机协同对模型可信度的重要性
时序分析的独特之处在于时间表示是实验设计的核心，需准确反映应用相关的时序关系。本文研究两种Transformer架构：一种采用文献中固定的时序表示，另一种让时序表示从数据中学习。实验基于太阳能发电量预测，该任务具有明确的昼夜和季节周期性，易于通过固定编码实现。结果表明，即使在已知规律清晰的情况下，固定编码仍难以有效融入先验知识，且伴随难以消除的副作用。研究呼吁未来工作应将人类纳入学习闭环，以提升模型的鲁棒性与可信度。
原文摘要 · Abstract (English)
What sets timeseries analysis apart from other machine learning exercises is that time representation becomes a primary aspect of the experiment setup, as it must adequately represent the temporal relations that are relevant for the application at hand. In the work described here we study wo different variations of the Transformer architecture: one where we use the fixed time representation proposed in the literature and one where the time representation is learned from the data. Our experiments use data from predicting the energy output of solar panels, a task that exhibits known periodicities (daily and seasonal) that is straight-forward to encode in the fixed time representation. Our results indicate that even in an experiment where the phenomenon is well-understood, it is difficult to encode prior knowledge due to side-effects that are difficult to mitigate. We conclude that research work is needed to work the human into the learning loop in ways that improve the robustness and trust-worthiness of the network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。