用线性时间编码替代正弦编码,更高效且性能不降反升。
Between Linear and Sinusoidal: Rethinking the Time Encoder in Dynamic Graph Learning
- 用线性编码代替正弦编码,避免时间信息丢失。
- 2维线性编码比100维正弦编码少43%参数,5个数据集表现更好。
- 适合追求轻量化与高效率的动态图模型设计者。
动态图学习对时序网络应用至关重要,需有效建模时间关系。主流注意力模型如TGAT和DyGFormer依赖正弦时间编码来捕捉边事件间的时间依赖。此前研究认为正弦编码因内积依赖事件间隔而适用,但其多对一特性会损失时间信息,需高维表示。本文系统研究了更简单的线性时间编码:它避免正弦函数带来的信息损失,减少对高维编码的需求。实验表明,自注意力机制能从线性编码中有效学习事件间隔并提取时间模式。在六个动态图数据集上,线性编码在多数情况下提升TGAT和DyGFormer性能;例如,相比100维正弦编码,2维线性编码使TGAT参数减少43%,并在五个数据集上实现更高平均精度。两者可共用,但本研究突显了线性时间特征在现代动态图模型中的被低估优势。这些发现将影响动态图架构设计,助力推荐系统、通信网络与交通预测等时序网络应用。
原文摘要 · Abstract (English)
Dynamic graph learning is essential for applications involving temporal networks and requires effective modeling of temporal relationships. Seminal attention-based models like TGAT and DyGFormer rely on sinusoidal time encoders to capture temporal dependencies between edge events. Prior work justified sinusoidal encodings because their inner products depend on the time spans between events, which are crucial features for modeling inter-event relations. However, sinusoidal encodings inherently lose temporal information due to their many-to-one nature and therefore require high dimensions. In this paper, we rigorously study a simpler alternative: the linear time encoder, which avoids temporal information loss caused by sinusoidal functions and reduces the need for high-dimensional time encoders. We show that the self-attention mechanism can effectively learn to compute time spans between events from linear time encodings and extract relevant temporal patterns. Through extensive experiments on six dynamic graph datasets, we demonstrate that the linear time encoder improves the performance of TGAT and DyGFormer in most cases. Moreover, the linear time encoder can lead to significant savings in model parameters with minimal performance loss. For example, compared to a 100-dimensional sinusoidal time encoder, TGAT with a 2-dimensional linear time encoder saves 43% of parameters and achieves higher average precision on five datasets. While both encoders can be used simultaneously, our study highlights the often-overlooked advantages of linear time features in modern dynamic graph models. These findings can positively impact the design choices of various dynamic graph learning architectures and eventually benefit temporal network applications such as recommender systems, communication networks, and traffic forecasting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。